2024-03-25T16:24:38.694 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:24:38.694 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280 --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/orig.config.yaml 2024-03-25T16:24:38.737 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:24:38.814 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621280' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBAYvnjUY3nI+HIYO8Dn/SgNRNe9ZigdnjOdZ4vy2p9LJP7xZ4RUQ3G8wxeCTNr4a7yQkLUGc2eFGGr0tXpEf8= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDmwdMj0KKr+QmizJqJ9GC8PcQVl5kT7qN2x5x7cetOt//LXCqLl6za4k2jkJoSSdf8yvXr3HCWWda3EgwHDTI8= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:24:38.815 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:24:38.815 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:24:38.816 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:24:38.817 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:24:38.853 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:24:38.853 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:24:38.854 INFO:teuthology.packaging:ref: None 2024-03-25T16:24:38.854 INFO:teuthology.packaging:tag: None 2024-03-25T16:24:38.854 INFO:teuthology.packaging:branch: main 2024-03-25T16:24:38.854 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:24:38.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:24:39.205 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:24:39.207 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:24:39.230 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:24:39.230 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:24:39.235 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:24:39.251 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:24:39.258 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:24:39.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:19:17.529385', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBAYvnjUY3nI+HIYO8Dn/SgNRNe9ZigdnjOdZ4vy2p9LJP7xZ4RUQ3G8wxeCTNr4a7yQkLUGc2eFGGr0tXpEf8='} 2024-03-25T16:24:39.307 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:19:17.526797', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDmwdMj0KKr+QmizJqJ9GC8PcQVl5kT7qN2x5x7cetOt//LXCqLl6za4k2jkJoSSdf8yvXr3HCWWda3EgwHDTI8='} 2024-03-25T16:24:39.307 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:24:39.316 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:24:39.316 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:24:39.316 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:24:39.386 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f385751b0d0>, signals=[15]) 2024-03-25T16:24:39.387 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:24:39.399 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:24:39.399 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T16:24:39.401 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:39.501 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T16:24:39.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:39.589 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:24:39.601 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-03-25T16:24:39.618 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-03-25T16:24:39.618 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-03-25T16:24:39.674 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-03-25T16:24:39.674 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-03-25T16:24:39.674 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:24:39.675 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:24:39.676 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-03-25T16:24:39.694 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-03-25T16:24:39.711 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-03-25T16:24:39.711 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-03-25T16:24:39.771 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-03-25T16:24:39.771 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-03-25T16:24:39.771 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-03-25T16:24:39.771 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:24:39.771 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-03-25T16:24:39.771 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:24:39.771 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:24:39.771 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-03-25T16:24:39.772 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:24:39.772 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:24:39.772 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:24:39.772 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:24:39.772 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:24:39.772 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:24:39.772 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-03-25T16:24:39.794 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:24:39.805 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:24:39.811 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:24:39.811 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:24:39.814 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:24:39.830 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:24:39.836 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:24:39.837 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:24:39.869 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:24:39.974 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:24:40.036 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:24:40.062 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:24:40.062 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:24:40.063 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:24:40.063 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:24:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:24:40.079 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-25T16:24:40.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:24:40.080 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-25T16:24:40.136 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:24:40.136 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.14.0-361.el9.x86_64 2024-03-25T16:24:40.137 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-03-25T16:24:40.137 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:24:40.138 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-361.el9.x86_64 2024-03-25T16:24:40.138 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-03-25T16:24:42.182 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:24:42.300 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:24:43.574 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-25T16:24:44.009 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-25T16:24:44.548 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T16:24:44.841 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-25T16:24:46.566 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-25T16:24:46.658 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-25T16:24:50.634 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:24:52.127 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-03-25T16:24:53.142 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.1 MB 00:03 2024-03-25T16:24:55.419 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 21 MB 00:03 2024-03-25T16:24:55.569 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T16:25:00.361 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T16:25:02.392 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:25:02.393 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:25:02.492 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:25:02.505 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:25:02.506 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:25:02.506 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-03-25T16:25:02.506 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:25:02.506 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 76 M 2024-03-25T16:25:02.506 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 123 M 2024-03-25T16:25:02.506 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:25:03.262 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 101 kB/s | 44 kB 00:00 2024-03-25T16:25:09.928 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 2.1 MB/s | 15 MB 00:07 2024-03-25T16:25:13.269 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 2.7 MB/s | 27 MB 00:10 2024-03-25T16:25:16.136 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-25T16:25:21.033 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T16:25:23.148 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:25:23.148 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:25:23.250 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:25:23.254 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:25:23.254 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-25T16:25:23.254 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:25:23.257 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:25:23.257 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-03-25T16:25:23.258 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:25:23.259 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 76 M 2024-03-25T16:25:23.259 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 123 M 2024-03-25T16:25:23.259 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:25:23.726 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 128 kB/s | 44 kB 00:00 2024-03-25T16:25:28.943 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.3 MB/s | 33 MB 00:26 2024-03-25T16:25:28.943 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:28.944 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.9 MB/s | 76 MB 00:26 2024-03-25T16:25:29.608 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:25:29.971 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:25:29.971 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:25:30.459 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 4.7 MB/s | 33 MB 00:07 2024-03-25T16:25:30.990 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:25:30.991 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:25:33.201 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.5 MB/s | 15 MB 00:09 2024-03-25T16:25:33.487 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:25:34.356 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:25:34.451 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:35.951 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 2.2 MB/s | 27 MB 00:12 2024-03-25T16:25:35.951 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:35.951 INFO:teuthology.orchestra.run.smithi102.stdout:Total 6.0 MB/s | 76 MB 00:12 2024-03-25T16:25:36.611 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:25:36.961 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:25:36.961 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:25:37.186 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:37.318 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:37.962 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:25:37.963 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:25:40.446 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:25:41.376 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:42.025 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:42.118 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:44.118 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:25:44.247 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:46.854 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:48.974 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:25:49.014 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:25:53.838 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:15.976 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:15.977 INFO:teuthology.orchestra.run.smithi146.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:26:15.977 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:26:16.020 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:26:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:26:16.399 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:26:16.821 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:26:17.191 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:17.295 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:17.295 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:17.295 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:26:17.295 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:26:17.295 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:26:17.296 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:26:17.296 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:26:17.296 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:17.376 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:17.376 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:17.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:26:17.415 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:26:17.416 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:26:17.417 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-03-25T16:26:17.417 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:26:17.417 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:26:17.417 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:26:17.418 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:26:17.418 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:26:18.014 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-03-25T16:26:19.368 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:26:19.389 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-03-25T16:26:19.394 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-25T16:26:19.410 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.m6TPKPPO1D 2024-03-25T16:26:19.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.m6TPKPPO1D 2024-03-25T16:26:19.479 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.m6TPKPPO1D 2024-03-25T16:26:19.615 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.m6TPKPPO1D is 6KB 2024-03-25T16:26:19.666 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.m6TPKPPO1D 2024-03-25T16:26:19.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:26:19.745 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:26:19.745 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:26:19.745 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:26:19.745 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:26:19.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:26:19.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-03-25T16:26:23.036 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:23.036 INFO:teuthology.orchestra.run.smithi102.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:26:23.037 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:26:23.081 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:23.496 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:23.496 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:26:23.497 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:26:23.497 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:26:23.926 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:26:23.926 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:26:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:26:23.927 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:23.927 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:23.927 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:26:23.927 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:26:23.927 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:26:23.927 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:26:24.250 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:24.343 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:24.344 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:24.344 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:26:24.344 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:26:24.344 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:26:24.344 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:26:24.344 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:26:24.344 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:26:24.424 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:26:24.425 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:26:24.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:26:24.459 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:26:24.460 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:26:24.461 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:26:24.461 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:26:24.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:26:25.067 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-03-25T16:26:26.390 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:26:26.416 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-03-25T16:26:26.418 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-25T16:26:26.433 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.s0ZbngjsHP 2024-03-25T16:26:26.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.s0ZbngjsHP 2024-03-25T16:26:26.503 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.s0ZbngjsHP 2024-03-25T16:26:26.653 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.s0ZbngjsHP is 6KB 2024-03-25T16:26:26.703 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.s0ZbngjsHP 2024-03-25T16:26:26.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:26:26.781 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:26:26.781 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:26:26.782 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:26:26.782 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:26:26.783 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:26:26.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-03-25T16:26:49.914 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:26:49.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-25T16:26:49.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:26:56.930 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:26:56.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-25T16:26:56.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:08.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-25T16:27:15.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-25T16:27:17.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-25T16:27:17.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:20.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-25T16:27:24.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-25T16:27:24.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:27.426 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-25T16:27:32.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-25T16:27:32.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:39.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-25T16:27:39.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:50.946 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-25T16:27:57.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-25T16:28:05.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-25T16:28:05.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:09.131 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-25T16:28:09.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-25T16:28:09.629 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:28:09.629 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-25T16:28:09.643 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:28:09.643 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:28:09.644 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:28:09.644 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:28:10.644 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:28:10.644 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:28:10.736 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-03-25T16:28:10.775 DEBUG:teuthology.parallel:result is None 2024-03-25T16:28:12.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-25T16:28:12.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:28:13.205 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-25T16:28:13.668 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-25T16:28:13.668 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:28:13.668 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-25T16:28:13.683 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:28:13.683 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:28:13.684 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:28:13.684 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:28:14.685 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:28:14.685 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:28:14.765 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-03-25T16:28:14.797 DEBUG:teuthology.parallel:result is None 2024-03-25T16:28:14.798 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:28:14.805 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:28:14.805 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:28:14.807 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:28:14.822 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:28:14.828 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:28:14.836 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:28:14.836 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:28:14.863 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:28:14.903 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:28:14.914 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:28:14.914 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:28:14.941 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:28:14.965 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:14.972 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:14.978 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:14.985 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:28:14.987 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:28:14.994 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:28:14.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:28:15.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:28:15.057 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:28:15.066 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:28:15.066 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:28:15.090 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:28:15.116 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:15.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:15.245 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:15.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:15.347 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T16:28:15.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:28:15.430 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:15.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:28:15.553 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:15.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:28:15.652 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T16:28:15.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:28:15.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-03-25T16:28:15.747 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-03-25T16:28:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:28:15.973 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:28:16.425 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:28:16.434 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:28:16.434 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:28:16.460 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:28:16.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-03-25T16:28:16.510 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-03-25T16:28:16.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-03-25T16:28:16.601 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-03-25T16:28:16.604 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:28:16.604 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-03-25T16:28:16.773 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-03-25T16:28:16.773 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-03-25T16:28:16.807 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-03-25T16:28:16.807 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:28:16.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:28:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:28:16.837 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:28:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:28:16.882 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:28:16.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:28:16.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:28:17.051 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:28:17.060 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:28:17.078 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:28:17.080 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:32:37.855 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-03-25T16:32:37.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-25T16:32:37.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:32:37.946 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-25T16:32:38.025 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-25T16:32:38.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-25T16:32:38.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:32:38.111 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-25T16:32:38.192 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-25T16:32:38.193 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:32:38.203 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:32:38.204 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:32:38.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:32:38.206 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:32:38.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:32:38.234 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:32:38.250 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:32:38.262 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:32:38.279 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:32:38.279 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-03-25T16:32:38.292 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-03-25T16:32:38.308 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-03-25T16:32:38.309 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:32:38.322 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-03-25T16:32:38.324 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:32:38.338 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:32:38.354 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:32:38.377 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:32:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:32:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-25T16:32:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.381 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.407 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:32:38.410 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:32:38.411 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-25T16:32:38.411 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.411 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.411 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.411 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:32:38.412 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:32:38.420 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:32:38.420 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:32:38.420 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:32:38.420 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:32:38.426 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:32:38.426 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:32:38.427 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:32:38.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:32:38.429 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:32:38.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:32:38.737 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:32:38.737 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:32:38.741 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:32:38.741 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:32:39.083 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:32:39.083 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T16:32:39.083 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:32:39.115 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:32:39.115 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:32:39.169 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:32:39.170 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T16:32:39.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:32:39.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:32:39.198 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:32:39.198 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:32:39.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:32:39.269 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:32:39.336 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-03-25T16:32:39.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-03-25T16:32:39.345 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:32:39.412 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-03-25T16:32:39.414 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-03-25T16:32:39.576 INFO:teuthology.orchestra.run.smithi146.stdout:74 files removed 2024-03-25T16:32:39.603 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw 2024-03-25T16:32:39.658 INFO:teuthology.orchestra.run.smithi102.stdout:74 files removed 2024-03-25T16:32:39.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw 2024-03-25T16:32:40.814 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 90 kB/s | 82 kB 00:00 2024-03-25T16:32:41.065 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 75 kB/s | 82 kB 00:01 2024-03-25T16:32:41.640 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:32:41.891 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:32:42.515 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:32:42.708 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:32:42.961 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T16:32:43.195 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:32:44.362 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-25T16:32:44.803 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-25T16:32:53.510 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 933 kB/s | 8.0 MB 00:08 2024-03-25T16:32:54.251 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 901 kB/s | 8.0 MB 00:09 2024-03-25T16:32:56.535 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-25T16:32:57.736 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 19 MB 00:03 2024-03-25T16:33:00.511 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T16:33:01.744 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-25T16:33:02.145 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-25T16:33:03.105 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-25T16:33:03.296 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-25T16:33:04.891 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T16:33:07.988 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:33:09.755 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-25T16:33:10.433 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:33:10.451 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:33:10.451 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:33:10.451 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:33:10.451 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:33:10.451 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:33:10.451 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:33:10.452 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:33:10.453 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout:Install 30 Packages 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 2 Packages 2024-03-25T16:33:10.454 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:33:10.455 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 58 M 2024-03-25T16:33:10.455 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:33:12.192 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:33:12.201 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:33:12.201 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:33:12.201 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:33:12.201 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:33:12.202 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:33:12.203 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout:Install 30 Packages 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 2 Packages 2024-03-25T16:33:12.204 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:33:12.205 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 58 M 2024-03-25T16:33:12.205 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:33:26.371 INFO:teuthology.orchestra.run.smithi146.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 392 kB/s | 5.3 MB 00:13 2024-03-25T16:33:26.471 INFO:teuthology.orchestra.run.smithi146.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:33:27.729 INFO:teuthology.orchestra.run.smithi146.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 589 kB/s | 742 kB 00:01 2024-03-25T16:33:28.780 INFO:teuthology.orchestra.run.smithi146.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 475 kB/s | 498 kB 00:01 2024-03-25T16:33:32.322 INFO:teuthology.orchestra.run.smithi102.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 253 kB/s | 5.3 MB 00:21 2024-03-25T16:33:32.771 INFO:teuthology.orchestra.run.smithi102.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 56 kB/s | 25 kB 00:00 2024-03-25T16:33:36.263 INFO:teuthology.orchestra.run.smithi102.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 212 kB/s | 742 kB 00:03 2024-03-25T16:33:38.022 INFO:teuthology.orchestra.run.smithi102.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 283 kB/s | 498 kB 00:01 2024-03-25T16:33:43.520 INFO:teuthology.orchestra.run.smithi146.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 354 kB/s | 5.1 MB 00:14 2024-03-25T16:33:43.704 INFO:teuthology.orchestra.run.smithi146.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 246 kB/s | 45 kB 00:00 2024-03-25T16:33:44.145 INFO:teuthology.orchestra.run.smithi146.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 297 kB/s | 131 kB 00:00 2024-03-25T16:33:44.604 INFO:teuthology.orchestra.run.smithi146.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 353 kB/s | 162 kB 00:00 2024-03-25T16:33:45.379 INFO:teuthology.orchestra.run.smithi146.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 413 kB/s | 320 kB 00:00 2024-03-25T16:33:46.021 INFO:teuthology.orchestra.run.smithi146.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 464 kB/s | 298 kB 00:00 2024-03-25T16:33:46.338 INFO:teuthology.orchestra.run.smithi146.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 314 kB/s | 99 kB 00:00 2024-03-25T16:33:46.730 INFO:teuthology.orchestra.run.smithi146.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 270 kB/s | 106 kB 00:00 2024-03-25T16:33:46.847 INFO:teuthology.orchestra.run.smithi146.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-25T16:33:46.905 INFO:teuthology.orchestra.run.smithi146.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 759 kB/s | 44 kB 00:00 2024-03-25T16:33:47.039 INFO:teuthology.orchestra.run.smithi146.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-25T16:33:47.106 INFO:teuthology.orchestra.run.smithi146.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-25T16:33:47.148 INFO:teuthology.orchestra.run.smithi146.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2024-03-25T16:33:47.196 INFO:teuthology.orchestra.run.smithi102.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 273 kB/s | 9.7 MB 00:36 2024-03-25T16:33:47.248 INFO:teuthology.orchestra.run.smithi146.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 2.1 MB/s | 216 kB 00:00 2024-03-25T16:33:47.290 INFO:teuthology.orchestra.run.smithi146.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 835 kB/s | 35 kB 00:00 2024-03-25T16:33:47.340 INFO:teuthology.orchestra.run.smithi146.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 892 kB/s | 44 kB 00:00 2024-03-25T16:33:47.371 INFO:teuthology.orchestra.run.smithi102.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 258 kB/s | 45 kB 00:00 2024-03-25T16:33:47.649 INFO:teuthology.orchestra.run.smithi146.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-25T16:33:47.788 INFO:teuthology.orchestra.run.smithi102.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 315 kB/s | 131 kB 00:00 2024-03-25T16:33:47.924 INFO:teuthology.orchestra.run.smithi146.stdout:(22/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 281 kB/s | 9.7 MB 00:35 2024-03-25T16:33:47.999 INFO:teuthology.orchestra.run.smithi146.stdout:(23/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-25T16:33:48.058 INFO:teuthology.orchestra.run.smithi146.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-03-25T16:33:48.125 INFO:teuthology.orchestra.run.smithi146.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-03-25T16:33:48.142 INFO:teuthology.orchestra.run.smithi146.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 805 kB/s | 67 kB 00:00 2024-03-25T16:33:48.205 INFO:teuthology.orchestra.run.smithi102.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 389 kB/s | 162 kB 00:00 2024-03-25T16:33:48.217 INFO:teuthology.orchestra.run.smithi146.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T16:33:48.334 INFO:teuthology.orchestra.run.smithi146.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-25T16:33:48.501 INFO:teuthology.orchestra.run.smithi146.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-25T16:33:49.314 INFO:teuthology.orchestra.run.smithi102.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 289 kB/s | 320 kB 00:01 2024-03-25T16:33:50.289 INFO:teuthology.orchestra.run.smithi102.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 305 kB/s | 298 kB 00:00 2024-03-25T16:33:50.714 INFO:teuthology.orchestra.run.smithi102.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 234 kB/s | 99 kB 00:00 2024-03-25T16:33:50.973 INFO:teuthology.orchestra.run.smithi102.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 410 kB/s | 106 kB 00:00 2024-03-25T16:33:51.081 INFO:teuthology.orchestra.run.smithi102.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-25T16:33:51.148 INFO:teuthology.orchestra.run.smithi102.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 664 kB/s | 44 kB 00:00 2024-03-25T16:33:51.290 INFO:teuthology.orchestra.run.smithi102.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2024-03-25T16:33:51.374 INFO:teuthology.orchestra.run.smithi102.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-25T16:33:51.441 INFO:teuthology.orchestra.run.smithi102.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 618 kB/s | 41 kB 00:00 2024-03-25T16:33:51.575 INFO:teuthology.orchestra.run.smithi102.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.6 MB/s | 216 kB 00:00 2024-03-25T16:33:51.616 INFO:teuthology.orchestra.run.smithi102.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 840 kB/s | 35 kB 00:00 2024-03-25T16:33:51.659 INFO:teuthology.orchestra.run.smithi102.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:33:52.084 INFO:teuthology.orchestra.run.smithi102.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 727 kB/s | 309 kB 00:00 2024-03-25T16:33:52.401 INFO:teuthology.orchestra.run.smithi102.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-25T16:33:52.476 INFO:teuthology.orchestra.run.smithi102.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-25T16:33:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2024-03-25T16:33:52.627 INFO:teuthology.orchestra.run.smithi102.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 898 kB/s | 67 kB 00:00 2024-03-25T16:33:52.719 INFO:teuthology.orchestra.run.smithi102.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-03-25T16:33:52.812 INFO:teuthology.orchestra.run.smithi102.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-25T16:33:52.911 INFO:teuthology.orchestra.run.smithi102.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-25T16:33:55.475 INFO:teuthology.orchestra.run.smithi146.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 484 kB/s | 3.4 MB 00:07 2024-03-25T16:33:59.911 INFO:teuthology.orchestra.run.smithi102.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 239 kB/s | 5.1 MB 00:21 2024-03-25T16:34:01.158 INFO:teuthology.orchestra.run.smithi146.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 253 kB/s | 3.1 MB 00:12 2024-03-25T16:34:04.669 INFO:teuthology.orchestra.run.smithi102.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 294 kB/s | 3.4 MB 00:11 2024-03-25T16:34:07.485 INFO:teuthology.orchestra.run.smithi102.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 423 kB/s | 3.1 MB 00:07 2024-03-25T16:34:09.693 INFO:teuthology.orchestra.run.smithi102.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 345 kB/s | 20 MB 00:59 2024-03-25T16:34:09.697 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:09.697 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.0 MB/s | 58 MB 00:59 2024-03-25T16:34:09.726 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:34:09.764 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:34:09.764 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:34:10.236 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:34:10.236 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:34:11.712 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:34:11.801 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:34:11.840 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:34:11.865 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:34:11.896 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:11.912 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:12.081 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:12.135 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:12.524 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:34:12.550 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:12.641 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:12.729 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:34:12.768 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:12.830 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:12.859 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:34:12.903 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:34:12.971 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:12.995 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:13.042 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:13.069 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:34:13.179 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:34:13.220 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:34:13.261 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:34:13.294 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:34:13.323 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:34:13.364 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:34:13.704 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:34:13.735 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:13.853 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:13.881 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:34:13.924 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:34:14.387 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:34:14.421 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:14.484 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:14.552 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:34:14.582 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:34:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:16.753 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:17.293 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:17.331 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:17.431 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:34:17.431 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:34:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:34:17.466 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:23.147 INFO:teuthology.orchestra.run.smithi146.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 288 kB/s | 20 MB 01:10 2024-03-25T16:34:23.150 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:23.150 INFO:teuthology.orchestra.run.smithi146.stdout:Total 835 kB/s | 58 MB 01:10 2024-03-25T16:34:23.181 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:34:23.219 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:34:23.219 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:34:23.697 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:34:23.698 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:34:25.189 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:34:25.274 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:34:25.302 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:34:25.327 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:34:25.349 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:25.366 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:34:25.557 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:25.614 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:34:25.940 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:34:25.995 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:26.096 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:26.184 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:34:26.236 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:26.292 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:34:26.321 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:34:26.366 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:34:26.435 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:26.457 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:26.520 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:34:26.547 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:34:26.660 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:34:26.699 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:34:26.739 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:34:26.771 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:34:26.800 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-03-25T16:34:26.819 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:34:26.833 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:34:27.155 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:34:27.179 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:27.252 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:34:27.284 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:34:27.329 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:34:27.750 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:27.773 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:27.773 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:27.773 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:34:27.773 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:27.773 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:27.773 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:34:27.775 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:27.785 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:34:27.823 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:27.860 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:27.860 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:27.894 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:34:27.962 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:34:27.992 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:34:28.984 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:28.984 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:34:28.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:34:28.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:34:28.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:34:28.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:28.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:34:28.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:34:30.059 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:30.059 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:34:30.059 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-03-25T16:34:30.059 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.059 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:34:30.060 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:30.061 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:34:30.062 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:34:30.062 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:34:30.062 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:34:30.062 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:34:30.062 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:34:30.062 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:34:30.134 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:30.180 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:30.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test 2024-03-25T16:34:30.733 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:34:30.766 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:34:30.859 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:34:30.859 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:34:30.859 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:34:30.898 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:01:23 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:34:30.901 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:31.046 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:34:31.047 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:34:31.047 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:34:31.047 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-03-25T16:34:31.048 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:34:31.049 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 48 M 2024-03-25T16:34:31.049 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 204 M 2024-03-25T16:34:31.049 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:34:31.322 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T16:34:31.372 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:34:31.405 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:34:31.681 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 683 kB/s | 187 kB 00:00 2024-03-25T16:34:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 150 kB/s | 163 kB 00:01 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-03-25T16:34:40.259 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:34:41.157 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:34:41.181 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:41.181 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:34:41.181 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:41.181 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:34:41.181 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:34:41.183 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:41.267 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:34:41.268 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:42.408 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:42.408 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:34:42.408 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:34:42.408 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:34:42.409 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:34:42.410 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:34:42.411 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:34:42.411 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:34:42.411 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:34:42.411 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:34:42.411 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:34:42.412 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:34:43.537 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:34:43.537 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:34:43.537 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-03-25T16:34:43.537 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:34:43.538 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:34:43.539 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:34:43.539 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.539 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:34:43.539 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.539 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:34:43.539 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:34:43.539 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:34:43.539 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:34:43.540 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:34:43.541 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:34:43.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test 2024-03-25T16:34:44.445 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:35 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:34:44.600 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:34:44.601 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:34:44.601 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:34:44.601 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:34:44.601 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:34:44.601 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-03-25T16:34:44.602 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:34:44.603 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 48 M 2024-03-25T16:34:44.603 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 204 M 2024-03-25T16:34:44.603 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:34:44.806 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T16:34:44.873 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.6 MB/s | 300 kB 00:00 2024-03-25T16:34:44.914 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T16:34:45.073 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T16:34:46.048 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 118 kB/s | 163 kB 00:01 2024-03-25T16:36:27.647 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 416 kB/s | 47 MB 01:56 2024-03-25T16:36:27.648 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:27.648 INFO:teuthology.orchestra.run.smithi102.stdout:Total 423 kB/s | 48 MB 01:56 2024-03-25T16:36:27.659 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:36:27.678 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:36:27.678 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:36:28.056 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:36:28.057 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:36:28.653 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:36:28.737 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:36:28.775 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:36:28.816 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:36:28.861 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:36:28.882 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:36:33.396 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:36:33.472 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:36:35.778 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:36:35.778 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:36:35.778 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:36:35.778 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:36:35.778 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:36:35.778 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:36:36.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:36:36.191 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:36:36.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph 2024-03-25T16:36:37.047 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:36:37.201 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================= 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================= 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:36:37.204 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:36:37.205 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:36:37.206 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout:========================================================================================= 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout:Install 36 Packages 2024-03-25T16:36:37.207 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:36:37.208 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 30 M 2024-03-25T16:36:37.208 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 107 M 2024-03-25T16:36:37.208 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:36:37.758 INFO:teuthology.orchestra.run.smithi102.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:36:43.216 INFO:teuthology.orchestra.run.smithi102.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 262 kB/s | 1.5 MB 00:05 2024-03-25T16:36:44.449 INFO:teuthology.orchestra.run.smithi102.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 308 kB/s | 2.1 MB 00:07 2024-03-25T16:36:45.050 INFO:teuthology.orchestra.run.smithi102.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 412 kB/s | 247 kB 00:00 2024-03-25T16:36:45.167 INFO:teuthology.orchestra.run.smithi102.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:36:45.209 INFO:teuthology.orchestra.run.smithi102.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:36:45.242 INFO:teuthology.orchestra.run.smithi102.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-25T16:36:45.276 INFO:teuthology.orchestra.run.smithi102.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:36:45.318 INFO:teuthology.orchestra.run.smithi102.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 924 kB/s | 39 kB 00:00 2024-03-25T16:36:45.360 INFO:teuthology.orchestra.run.smithi102.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:36:45.435 INFO:teuthology.orchestra.run.smithi102.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:36:45.510 INFO:teuthology.orchestra.run.smithi102.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-25T16:36:45.619 INFO:teuthology.orchestra.run.smithi102.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 1.1 MB/s | 124 kB 00:00 2024-03-25T16:36:45.661 INFO:teuthology.orchestra.run.smithi102.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:36:46.444 INFO:teuthology.orchestra.run.smithi102.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 159 kB/s | 125 kB 00:00 2024-03-25T16:36:46.686 INFO:teuthology.orchestra.run.smithi102.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-03-25T16:36:46.861 INFO:teuthology.orchestra.run.smithi102.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 864 kB/s | 151 kB 00:00 2024-03-25T16:36:46.912 INFO:teuthology.orchestra.run.smithi102.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 868 kB/s | 43 kB 00:00 2024-03-25T16:36:46.979 INFO:teuthology.orchestra.run.smithi102.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-25T16:36:47.046 INFO:teuthology.orchestra.run.smithi102.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-25T16:36:47.088 INFO:teuthology.orchestra.run.smithi102.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 258 kB/s | 11 kB 00:00 2024-03-25T16:36:47.130 INFO:teuthology.orchestra.run.smithi102.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-03-25T16:36:47.172 INFO:teuthology.orchestra.run.smithi102.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 557 kB/s | 23 kB 00:00 2024-03-25T16:36:47.214 INFO:teuthology.orchestra.run.smithi102.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-03-25T16:36:47.256 INFO:teuthology.orchestra.run.smithi102.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-03-25T16:36:47.298 INFO:teuthology.orchestra.run.smithi102.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-25T16:36:47.339 INFO:teuthology.orchestra.run.smithi102.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2024-03-25T16:36:47.390 INFO:teuthology.orchestra.run.smithi102.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T16:36:47.432 INFO:teuthology.orchestra.run.smithi102.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 393 kB/s | 16 kB 00:00 2024-03-25T16:36:47.482 INFO:teuthology.orchestra.run.smithi102.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-25T16:36:47.524 INFO:teuthology.orchestra.run.smithi102.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 861 kB/s | 36 kB 00:00 2024-03-25T16:36:47.574 INFO:teuthology.orchestra.run.smithi102.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T16:36:47.624 INFO:teuthology.orchestra.run.smithi102.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.4 MB/s | 427 kB 00:00 2024-03-25T16:36:47.666 INFO:teuthology.orchestra.run.smithi102.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-03-25T16:36:49.066 INFO:teuthology.orchestra.run.smithi102.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 424 kB/s | 4.7 MB 00:11 2024-03-25T16:37:38.942 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 278 kB/s | 47 MB 02:54 2024-03-25T16:37:38.943 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:38.944 INFO:teuthology.orchestra.run.smithi146.stdout:Total 283 kB/s | 48 MB 02:54 2024-03-25T16:37:38.956 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:37:38.977 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:37:38.977 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:37:39.360 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:37:39.360 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:37:39.951 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:37:40.033 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:37:40.081 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:37:40.122 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:37:40.184 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:37:40.211 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:37:44.793 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:37:44.868 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:37:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:37:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:37:47.182 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:37:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:37:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:37:47.183 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:37:47.549 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:37:47.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph 2024-03-25T16:37:48.466 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:37:48.625 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:37:48.627 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:37:48.628 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:37:48.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:37:48.630 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:37:48.631 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:37:48.631 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:37:48.631 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:37:48.631 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-03-25T16:37:48.631 INFO:teuthology.orchestra.run.smithi146.stdout:Install 36 Packages 2024-03-25T16:37:48.631 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:37:48.631 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 30 M 2024-03-25T16:37:48.632 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 107 M 2024-03-25T16:37:48.632 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:37:49.779 INFO:teuthology.orchestra.run.smithi146.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T16:37:51.585 INFO:teuthology.orchestra.run.smithi102.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 251 kB/s | 17 MB 01:08 2024-03-25T16:37:51.589 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:51.590 INFO:teuthology.orchestra.run.smithi102.stdout:Total 408 kB/s | 30 MB 01:14 2024-03-25T16:37:51.610 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:37:51.651 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:37:51.651 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:37:52.065 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:37:52.065 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:37:53.406 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:37:53.462 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:37:53.496 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:37:53.531 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:37:53.587 INFO:teuthology.orchestra.run.smithi146.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 506 kB/s | 2.1 MB 00:04 2024-03-25T16:37:53.592 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:37:53.639 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:37:53.671 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:37:53.761 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:37:53.832 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:37:53.882 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:37:53.953 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:37:54.016 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:37:54.054 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:37:54.103 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:37:54.166 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:37:54.349 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:37:54.418 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:37:54.456 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:37:54.490 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:37:54.521 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:37:54.596 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:37:54.631 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:37:54.656 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:37:54.696 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:37:54.731 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:37:54.766 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:37:54.855 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:37:54.897 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:37:54.968 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:37:55.105 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:37:55.220 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:37:55.379 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:37:55.402 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:37:55.427 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:37:55.427 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:55.427 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:37:55.427 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:37:55.427 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:37:55.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:37:55.795 INFO:teuthology.orchestra.run.smithi146.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 234 kB/s | 1.5 MB 00:06 2024-03-25T16:37:56.545 INFO:teuthology.orchestra.run.smithi146.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 330 kB/s | 247 kB 00:00 2024-03-25T16:37:56.662 INFO:teuthology.orchestra.run.smithi146.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:37:56.721 INFO:teuthology.orchestra.run.smithi146.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T16:37:56.754 INFO:teuthology.orchestra.run.smithi146.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-25T16:37:56.796 INFO:teuthology.orchestra.run.smithi146.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T16:37:56.838 INFO:teuthology.orchestra.run.smithi146.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 930 kB/s | 39 kB 00:00 2024-03-25T16:37:56.895 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:37:56.896 INFO:teuthology.orchestra.run.smithi146.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T16:37:56.923 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:37:56.923 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:56.923 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:37:56.923 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:37:56.923 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:37:56.923 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:37:57.005 INFO:teuthology.orchestra.run.smithi146.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-25T16:37:57.047 INFO:teuthology.orchestra.run.smithi146.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:37:57.080 INFO:teuthology.orchestra.run.smithi146.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T16:37:57.122 INFO:teuthology.orchestra.run.smithi146.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:37:57.364 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:37:57.386 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:37:57.386 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:57.387 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:37:57.387 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:37:57.387 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:37:57.387 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:37:57.598 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:37:57.621 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:37:57.621 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:37:57.621 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:37:57.621 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:37:57.621 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:37:57.621 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:37:57.686 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:37:57.923 INFO:teuthology.orchestra.run.smithi146.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 156 kB/s | 125 kB 00:00 2024-03-25T16:37:58.015 INFO:teuthology.orchestra.run.smithi146.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-03-25T16:37:58.107 INFO:teuthology.orchestra.run.smithi146.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-25T16:37:58.140 INFO:teuthology.orchestra.run.smithi146.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-25T16:37:58.190 INFO:teuthology.orchestra.run.smithi146.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-03-25T16:37:58.241 INFO:teuthology.orchestra.run.smithi146.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 7.0 MB/s | 358 kB 00:00 2024-03-25T16:37:58.275 INFO:teuthology.orchestra.run.smithi146.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 321 kB/s | 11 kB 00:00 2024-03-25T16:37:58.308 INFO:teuthology.orchestra.run.smithi146.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 534 kB/s | 18 kB 00:00 2024-03-25T16:37:58.342 INFO:teuthology.orchestra.run.smithi146.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 696 kB/s | 23 kB 00:00 2024-03-25T16:37:58.375 INFO:teuthology.orchestra.run.smithi146.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 586 kB/s | 19 kB 00:00 2024-03-25T16:37:58.409 INFO:teuthology.orchestra.run.smithi146.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 592 kB/s | 20 kB 00:00 2024-03-25T16:37:58.442 INFO:teuthology.orchestra.run.smithi146.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-03-25T16:37:58.468 INFO:teuthology.orchestra.run.smithi146.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 3.1 MB/s | 79 kB 00:00 2024-03-25T16:37:58.501 INFO:teuthology.orchestra.run.smithi146.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-03-25T16:37:58.535 INFO:teuthology.orchestra.run.smithi146.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 495 kB/s | 16 kB 00:00 2024-03-25T16:37:58.569 INFO:teuthology.orchestra.run.smithi146.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-03-25T16:37:58.594 INFO:teuthology.orchestra.run.smithi146.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-03-25T16:37:58.628 INFO:teuthology.orchestra.run.smithi146.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 6.7 MB/s | 230 kB 00:00 2024-03-25T16:37:58.670 INFO:teuthology.orchestra.run.smithi146.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 10 MB/s | 427 kB 00:00 2024-03-25T16:37:58.695 INFO:teuthology.orchestra.run.smithi146.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 805 kB/s | 20 kB 00:00 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:37:59.357 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:37:59.358 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:37:59.362 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:37:59.363 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:37:59.364 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:37:59.364 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:37:59.364 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:37:59.364 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:37:59.364 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:37:59.364 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:37:59.364 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:38:00.289 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:38:00.290 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:00.290 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:38:00.290 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:00.290 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:38:00.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:38:00.292 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:38:00.292 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:38:00.292 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:38:00.292 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:38:00.292 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:38:00.292 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:38:00.293 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:00.294 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:38:00.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base 2024-03-25T16:38:01.163 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:04:54 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:38:01.259 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:38:01.311 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:38:01.392 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install cephadm 2024-03-25T16:38:01.946 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:04:54 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:38:02.094 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 747 k 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 754 k 2024-03-25T16:38:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:38:06.629 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 165 kB/s | 747 kB 00:04 2024-03-25T16:38:06.630 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:06.630 INFO:teuthology.orchestra.run.smithi102.stdout:Total 165 kB/s | 747 kB 00:04 2024-03-25T16:38:06.631 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:38:06.636 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:38:06.636 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:38:06.642 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:38:06.643 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:38:06.714 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:38:07.130 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:07.218 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:07.616 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:07.903 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:07.903 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:07.903 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:38:07.903 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:07.904 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:07.904 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:38:08.115 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:38:08.691 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:38:08.845 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 145 k 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 436 k 2024-03-25T16:38:08.846 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:38:09.586 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 197 kB/s | 145 kB 00:00 2024-03-25T16:38:09.587 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:09.587 INFO:teuthology.orchestra.run.smithi102.stdout:Total 196 kB/s | 145 kB 00:00 2024-03-25T16:38:09.587 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:38:09.594 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:38:09.594 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:38:09.647 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:38:09.647 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:38:09.794 INFO:teuthology.orchestra.run.smithi146.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 240 kB/s | 4.7 MB 00:20 2024-03-25T16:38:09.794 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:38:09.862 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:09.886 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:09.886 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:09.886 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:38:09.886 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:10.856 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:38:10.904 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:10.905 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:38:10.905 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:10.905 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:10.905 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:38:11.098 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr 2024-03-25T16:38:11.664 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:05:04 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:38:11.761 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:11.812 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:38:11.813 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:38:11.814 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:38:11.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:38:12.454 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:05:05 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:38:12.602 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:38:12.603 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-03-25T16:38:12.603 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:38:12.603 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-03-25T16:38:12.603 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:38:12.604 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:38:12.605 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:12.605 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:38:12.605 INFO:teuthology.orchestra.run.smithi102.stdout:====================================================================================== 2024-03-25T16:38:12.605 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-03-25T16:38:12.605 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:12.606 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 6.9 M 2024-03-25T16:38:12.606 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 26 M 2024-03-25T16:38:12.606 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:38:13.298 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 62 kB/s | 31 kB 00:00 2024-03-25T16:38:13.440 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T16:38:13.482 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T16:38:13.507 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 21 kB/s | 15 kB 00:00 2024-03-25T16:38:13.716 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-03-25T16:38:13.799 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:38:13.816 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-03-25T16:38:13.966 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-03-25T16:38:13.983 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 866 kB/s | 144 kB 00:00 2024-03-25T16:38:14.008 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 735 kB/s | 31 kB 00:00 2024-03-25T16:38:14.050 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T16:38:16.325 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 516 kB/s | 1.8 MB 00:03 2024-03-25T16:38:16.329 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:16.330 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.8 MB/s | 6.9 MB 00:03 2024-03-25T16:38:16.349 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:38:16.368 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:38:16.368 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:38:16.542 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:38:16.543 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:38:17.068 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:38:17.151 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:38:17.186 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:38:17.232 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:38:17.264 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:38:17.361 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:38:17.492 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:38:17.582 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:38:17.626 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:38:17.655 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:38:17.686 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:38:17.927 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:38:17.962 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:38:18.770 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:38:18.770 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:38:18.770 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:38:18.770 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:38:18.770 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:38:18.770 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:38:18.770 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:38:18.770 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:38:18.771 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:38:18.771 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:38:18.771 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:38:18.771 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:38:19.138 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:38:19.138 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:19.138 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:38:19.138 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:38:19.139 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:19.139 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:19.139 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:19.139 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:38:19.139 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:38:19.139 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:38:19.140 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:38:19.140 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:38:19.140 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:38:19.140 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:38:19.140 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:38:19.140 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:19.140 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:38:19.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:38:19.985 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:38:20.136 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:38:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout:Install 12 Packages 2024-03-25T16:38:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 37 M 2024-03-25T16:38:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 209 M 2024-03-25T16:38:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:38:20.649 INFO:teuthology.orchestra.run.smithi102.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 179 kB/s | 32 kB 00:00 2024-03-25T16:38:20.716 INFO:teuthology.orchestra.run.smithi102.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 258 kB/s | 17 kB 00:00 2024-03-25T16:38:20.799 INFO:teuthology.orchestra.run.smithi102.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 410 kB/s | 34 kB 00:00 2024-03-25T16:38:21.433 INFO:teuthology.orchestra.run.smithi102.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.5 MB/s | 4.7 MB 00:00 2024-03-25T16:38:21.483 INFO:teuthology.orchestra.run.smithi102.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:01 2024-03-25T16:38:21.608 INFO:teuthology.orchestra.run.smithi102.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-25T16:38:21.675 INFO:teuthology.orchestra.run.smithi102.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-25T16:38:21.801 INFO:teuthology.orchestra.run.smithi102.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.4 MB/s | 803 kB 00:00 2024-03-25T16:38:21.876 INFO:teuthology.orchestra.run.smithi102.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-03-25T16:38:22.043 INFO:teuthology.orchestra.run.smithi102.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-25T16:38:22.259 INFO:teuthology.orchestra.run.smithi102.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 644 kB/s | 245 kB 00:00 2024-03-25T16:38:41.648 INFO:teuthology.orchestra.run.smithi146.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 357 kB/s | 17 MB 00:48 2024-03-25T16:38:41.652 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:41.652 INFO:teuthology.orchestra.run.smithi146.stdout:Total 573 kB/s | 30 MB 00:53 2024-03-25T16:38:41.674 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:38:41.718 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:38:41.718 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:38:42.108 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:38:42.108 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:38:43.442 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:38:43.509 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:38:43.543 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:38:43.577 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:38:43.647 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:38:43.685 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:38:43.717 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:38:43.809 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:38:43.879 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:38:43.921 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:38:43.991 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:38:44.045 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:38:44.084 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:38:44.132 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:38:44.141 INFO:teuthology.orchestra.run.smithi102.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 320 kB/s | 7.4 MB 00:23 2024-03-25T16:38:44.144 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:44.144 INFO:teuthology.orchestra.run.smithi102.stdout:Total 1.5 MB/s | 37 MB 00:24 2024-03-25T16:38:44.185 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:38:44.187 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:38:44.210 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:38:44.210 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:38:44.372 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:38:44.438 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:38:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:38:44.503 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:38:44.532 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:38:44.569 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:38:44.570 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:38:44.609 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:38:44.673 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:38:44.676 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:38:44.716 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:38:44.751 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:38:44.811 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:38:44.899 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:38:44.941 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:38:45.013 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:38:45.150 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:38:45.271 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:38:45.275 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:38:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:38:45.378 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:38:45.442 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:38:45.471 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:38:45.472 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:38:45.496 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:38:45.497 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:45.497 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:38:45.497 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:38:45.497 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:38:45.497 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:45.506 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:38:45.683 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:38:45.784 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:38:45.797 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:38:46.453 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:38:46.568 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:38:46.973 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:38:47.005 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:38:47.005 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:47.005 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:38:47.005 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:38:47.006 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:38:47.006 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:47.449 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:38:47.472 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:38:47.472 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:47.472 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:38:47.472 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:38:47.472 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:38:47.472 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:47.691 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:38:47.713 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:38:47.713 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:38:47.714 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:38:47.714 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:38:47.714 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:38:47.714 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:47.788 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:38:47.798 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:38:48.818 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:38:48.848 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:38:49.480 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:38:49.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:38:49.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:38:49.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:38:49.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:38:49.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:38:49.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:38:49.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:38:49.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:38:49.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:38:49.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:38:49.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:38:49.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:38:49.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:38:49.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:38:49.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:38:49.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:38:49.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:38:49.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:38:50.598 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:38:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:50.600 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:38:50.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:38:51.349 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:38:51.350 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:38:51.350 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:38:51.350 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:38:51.526 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:05:42 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:38:51.624 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:38:51.676 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:38:51.676 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:38:51.676 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:38:51.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install cephadm 2024-03-25T16:38:51.829 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:38:51.830 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:38:51.831 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:38:51.831 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:51.831 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:38:52.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:38:52.302 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:05:43 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:38:52.451 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:38:52.452 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:38:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:38:52.452 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:38:52.452 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:38:52.452 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:38:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:52.453 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:38:52.453 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:38:52.453 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T16:38:52.453 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:52.453 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 747 k 2024-03-25T16:38:52.453 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 754 k 2024-03-25T16:38:52.453 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:38:52.843 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:38:52.993 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:38:52.994 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T16:38:52.994 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:38:52.994 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T16:38:52.994 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:38:52.994 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:38:52.995 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:38:52.996 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:38:52.996 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:52.996 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:38:52.996 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T16:38:52.996 INFO:teuthology.orchestra.run.smithi102.stdout:Install 13 Packages 2024-03-25T16:38:52.996 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:52.997 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.0 M 2024-03-25T16:38:52.997 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 26 M 2024-03-25T16:38:52.997 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:38:53.239 INFO:teuthology.orchestra.run.smithi102.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 233 kB/s | 24 kB 00:00 2024-03-25T16:38:53.290 INFO:teuthology.orchestra.run.smithi102.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-25T16:38:53.340 INFO:teuthology.orchestra.run.smithi102.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:38:53.365 INFO:teuthology.orchestra.run.smithi102.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:38:53.390 INFO:teuthology.orchestra.run.smithi102.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:38:53.416 INFO:teuthology.orchestra.run.smithi102.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2024-03-25T16:38:53.466 INFO:teuthology.orchestra.run.smithi102.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 299 kB/s | 15 kB 00:00 2024-03-25T16:38:53.491 INFO:teuthology.orchestra.run.smithi102.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-03-25T16:38:53.541 INFO:teuthology.orchestra.run.smithi102.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.6 MB/s | 203 kB 00:00 2024-03-25T16:38:53.583 INFO:teuthology.orchestra.run.smithi102.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-25T16:38:53.617 INFO:teuthology.orchestra.run.smithi102.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-03-25T16:38:53.642 INFO:teuthology.orchestra.run.smithi102.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-25T16:38:53.767 INFO:teuthology.orchestra.run.smithi102.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 82 kB/s | 52 kB 00:00 2024-03-25T16:38:53.769 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:53.769 INFO:teuthology.orchestra.run.smithi102.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-25T16:38:53.770 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:38:53.783 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:38:53.783 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:38:53.874 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:38:53.881 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:38:54.395 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:38:54.564 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:38:54.628 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:38:54.684 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:38:54.720 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:38:54.762 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:38:54.859 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:38:54.967 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:38:55.022 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:38:55.775 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:38:55.818 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:38:55.852 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:38:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:38:55.930 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:38:56.745 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:38:56.745 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:38:56.745 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:38:56.746 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:38:57.092 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:38:57.093 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:38:57.094 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:38:57.094 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:38:57.094 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:57.094 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:38:57.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:38:57.779 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 140 kB/s | 747 kB 00:05 2024-03-25T16:38:57.780 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:57.781 INFO:teuthology.orchestra.run.smithi146.stdout:Total 140 kB/s | 747 kB 00:05 2024-03-25T16:38:57.781 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:38:57.787 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:38:57.787 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:38:57.793 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:38:57.794 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:38:57.874 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:38:57.947 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:38:58.097 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:38:58.098 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:38:58.099 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:38:58.099 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:58.099 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:38:58.099 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T16:38:58.099 INFO:teuthology.orchestra.run.smithi102.stdout:Install 6 Packages 2024-03-25T16:38:58.099 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:38:58.099 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 6.8 M 2024-03-25T16:38:58.100 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 33 M 2024-03-25T16:38:58.100 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:38:58.267 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:58.352 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:58.470 INFO:teuthology.orchestra.run.smithi102.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T16:38:58.701 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:58.778 INFO:teuthology.orchestra.run.smithi102.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-03-25T16:38:58.853 INFO:teuthology.orchestra.run.smithi102.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 769 kB/s | 58 kB 00:00 2024-03-25T16:38:58.905 INFO:teuthology.orchestra.run.smithi102.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-25T16:38:58.987 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:38:58.987 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:58.987 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:38:58.987 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:38:58.987 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:58.987 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:38:59.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:38:59.220 INFO:teuthology.orchestra.run.smithi102.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 246 kB/s | 248 kB 00:01 2024-03-25T16:38:59.262 INFO:teuthology.orchestra.run.smithi102.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 131 kB/s | 138 kB 00:01 2024-03-25T16:38:59.264 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:38:59.265 INFO:teuthology.orchestra.run.smithi102.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-25T16:38:59.278 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:38:59.289 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:38:59.290 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:38:59.379 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:38:59.379 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:38:59.680 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:38:59.789 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:38:59.850 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:38:59.898 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:38:59.937 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:38:59.937 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:38:59.937 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T16:38:59.937 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:38:59.937 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:38:59.937 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:38:59.938 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:59.938 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:38:59.938 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:38:59.938 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T16:38:59.938 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:38:59.938 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 145 k 2024-03-25T16:38:59.938 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 436 k 2024-03-25T16:38:59.939 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:39:00.408 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:39:00.501 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:39:00.562 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:39:00.585 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:39:01.101 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 125 kB/s | 145 kB 00:01 2024-03-25T16:39:01.102 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:01.102 INFO:teuthology.orchestra.run.smithi146.stdout:Total 125 kB/s | 145 kB 00:01 2024-03-25T16:39:01.102 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:39:01.112 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:39:01.112 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:39:01.168 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:39:01.169 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:39:01.312 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:39:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:39:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:39:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:39:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:39:01.382 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:39:01.383 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:39:01.385 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:39:01.411 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:39:01.411 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:39:01.411 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:39:01.411 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:01.674 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:39:01.674 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:01.674 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:39:01.674 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:01.674 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:39:01.674 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:39:01.674 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:39:01.675 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:39:01.675 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:39:01.675 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:01.675 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:01.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse 2024-03-25T16:39:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:39:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:02.407 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:39:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:02.407 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:02.407 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:39:02.498 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:02.650 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 964 k 2024-03-25T16:39:02.651 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 2.8 M 2024-03-25T16:39:02.652 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:39:02.687 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr 2024-03-25T16:39:02.772 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 651 kB/s | 79 kB 00:00 2024-03-25T16:39:03.255 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:05:54 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:39:03.352 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:03.403 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:39:03.404 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:39:03.404 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:39:03.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:39:04.037 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:39:04.189 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:39:04.190 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-25T16:39:04.190 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:39:04.190 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-25T16:39:04.190 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:39:04.190 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:39:04.191 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:04.192 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:39:04.192 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-03-25T16:39:04.192 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-03-25T16:39:04.192 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:04.192 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.9 M 2024-03-25T16:39:04.192 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-03-25T16:39:04.192 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:39:04.729 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 34 kB/s | 15 kB 00:00 2024-03-25T16:39:04.813 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-25T16:39:04.913 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-25T16:39:04.939 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 238 kB 00:00 2024-03-25T16:39:05.206 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 346 kB/s | 885 kB 00:02 2024-03-25T16:39:05.207 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:05.208 INFO:teuthology.orchestra.run.smithi102.stdout:Total 377 kB/s | 964 kB 00:02 2024-03-25T16:39:05.208 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:39:05.230 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:39:05.230 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:39:05.297 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T16:39:05.306 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:39:05.307 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:39:05.364 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-25T16:39:05.486 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:39:05.531 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.4 MB/s | 863 kB 00:00 2024-03-25T16:39:05.624 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:39:05.673 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 6.6 MB/s | 2.0 MB 00:00 2024-03-25T16:39:05.698 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:39:05.723 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 618 kB/s | 31 kB 00:00 2024-03-25T16:39:05.782 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-03-25T16:39:05.815 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 508 kB/s | 144 kB 00:00 2024-03-25T16:39:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:39:06.365 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:39:06.707 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:39:06.707 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:06.707 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:39:06.707 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:39:06.707 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:06.707 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:06.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-volume 2024-03-25T16:39:07.572 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:07.726 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:07.727 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:07.727 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-25T16:39:07.727 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:07.727 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 332 k 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 1.5 M 2024-03-25T16:39:07.728 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:39:07.838 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 629 kB/s | 69 kB 00:00 2024-03-25T16:39:09.430 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 155 kB/s | 264 kB 00:01 2024-03-25T16:39:09.431 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:09.432 INFO:teuthology.orchestra.run.smithi102.stdout:Total 195 kB/s | 332 kB 00:01 2024-03-25T16:39:09.432 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:39:09.441 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:39:09.442 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:39:09.481 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:39:09.482 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:39:09.634 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:39:09.819 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:39:09.849 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:39:09.864 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:39:09.864 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:39:09.864 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:39:09.864 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:10.595 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:39:10.957 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:39:10.957 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:10.957 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:39:10.958 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:10.958 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:39:10.958 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:10.958 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:11.073 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 268 kB/s | 1.8 MB 00:06 2024-03-25T16:39:11.078 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:11.078 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.0 MB/s | 6.9 MB 00:06 2024-03-25T16:39:11.098 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:39:11.116 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:39:11.117 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:39:11.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel 2024-03-25T16:39:11.300 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:39:11.301 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:39:11.833 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:39:11.856 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:11.915 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:39:11.950 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:39:11.997 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:39:12.005 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:12.005 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 127 k 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 456 k 2024-03-25T16:39:12.006 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:39:12.027 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:39:12.125 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:39:12.252 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:39:12.348 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:39:12.399 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:39:12.427 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:39:12.458 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:39:12.701 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:39:12.742 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:39:12.919 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 139 kB/s | 127 kB 00:00 2024-03-25T16:39:12.920 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:12.920 INFO:teuthology.orchestra.run.smithi102.stdout:Total 138 kB/s | 127 kB 00:00 2024-03-25T16:39:12.920 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:39:12.926 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:39:12.926 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:39:12.985 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:39:12.985 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:39:13.128 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:39:13.234 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:13.483 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:39:13.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:39:13.682 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:13.852 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:39:13.852 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:13.852 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:39:13.852 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:39:13.852 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:13.852 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:13.852 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:13.853 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:39:13.964 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:13.964 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:13.964 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:39:13.964 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:13.964 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:13.964 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:14.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:39:14.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2 2024-03-25T16:39:14.681 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:06:05 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:39:14.796 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:07 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:14.832 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:39:14.833 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-25T16:39:14.833 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:39:14.833 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-25T16:39:14.833 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:39:14.834 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:39:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:14.835 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:39:14.835 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-25T16:39:14.835 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-03-25T16:39:14.835 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:14.835 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 37 M 2024-03-25T16:39:14.835 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 209 M 2024-03-25T16:39:14.835 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:39:14.895 INFO:teuthology.orchestra.run.smithi102.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:14.948 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:14.949 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:39:14.949 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:15.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs-devel 2024-03-25T16:39:15.154 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 233 kB/s | 32 kB 00:00 2024-03-25T16:39:15.188 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 515 kB/s | 17 kB 00:00 2024-03-25T16:39:15.230 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 824 kB/s | 34 kB 00:00 2024-03-25T16:39:15.472 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-25T16:39:15.590 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:08 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:15.672 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-25T16:39:15.739 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-25T16:39:15.740 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T16:39:15.741 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:15.742 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 31 k 2024-03-25T16:39:15.742 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 140 k 2024-03-25T16:39:15.742 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:39:15.764 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-25T16:39:16.181 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-25T16:39:16.206 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-25T16:39:16.281 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 58 kB/s | 31 kB 00:00 2024-03-25T16:39:16.282 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:16.282 INFO:teuthology.orchestra.run.smithi102.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T16:39:16.282 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:39:16.286 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:39:16.286 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:39:16.318 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:39:16.318 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:39:16.331 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-25T16:39:16.356 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-25T16:39:16.448 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:39:16.564 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:16.949 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:17.271 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:17.272 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:17.272 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:39:17.272 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:17.272 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:17.272 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:17.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2 2024-03-25T16:39:18.004 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:11 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:18.101 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:18.153 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:18.154 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:39:18.154 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:18.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1 2024-03-25T16:39:18.770 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:11 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:18.867 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:18.918 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:18.919 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:39:18.919 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:18.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados 2024-03-25T16:39:19.534 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:12 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:19.636 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:19.691 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:19.692 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:39:19.692 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:19.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw 2024-03-25T16:39:20.343 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:20.443 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:20.496 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:20.497 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:39:20.497 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:20.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs 2024-03-25T16:39:21.114 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:21.209 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:21.260 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:21.261 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:39:21.261 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:21.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd 2024-03-25T16:39:21.877 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:21.973 INFO:teuthology.orchestra.run.smithi102.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:39:22.025 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:22.026 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T16:39:22.026 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:22.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse 2024-03-25T16:39:22.645 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:22.794 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:22.795 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:22.795 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 85 k 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 231 k 2024-03-25T16:39:22.796 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:39:23.494 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 122 kB/s | 85 kB 00:00 2024-03-25T16:39:23.494 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:23.495 INFO:teuthology.orchestra.run.smithi102.stdout:Total 122 kB/s | 85 kB 00:00 2024-03-25T16:39:23.495 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:39:23.502 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:39:23.502 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:39:23.558 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:39:23.558 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:39:23.690 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:39:23.801 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:24.211 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:24.501 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:24.501 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:24.501 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:39:24.501 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:24.501 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:24.501 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:24.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror 2024-03-25T16:39:25.345 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:18 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:25.499 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:25.499 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:25.499 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T16:39:25.499 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 3.1 M 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 13 M 2024-03-25T16:39:25.500 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:39:32.255 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 468 kB/s | 3.1 MB 00:06 2024-03-25T16:39:32.257 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:32.257 INFO:teuthology.orchestra.run.smithi102.stdout:Total 468 kB/s | 3.1 MB 00:06 2024-03-25T16:39:32.257 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:39:32.267 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:39:32.267 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:39:32.342 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:39:32.345 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:39:32.788 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:39:32.843 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:32.863 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:32.863 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:39:32.863 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:39:32.863 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:39:32.864 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:39:32.864 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:33.996 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:33.996 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:33.996 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:39:33.996 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:33.996 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:33.996 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:34.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd 2024-03-25T16:39:34.819 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:27 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:34.969 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:34.970 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:34.970 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-03-25T16:39:34.970 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:34.970 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:39:34.970 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:39:34.970 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:34.971 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:39:34.971 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:34.971 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T16:39:34.971 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:34.971 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 171 k 2024-03-25T16:39:34.971 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 494 k 2024-03-25T16:39:34.971 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:39:35.983 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 169 kB/s | 171 kB 00:01 2024-03-25T16:39:35.984 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:35.984 INFO:teuthology.orchestra.run.smithi102.stdout:Total 169 kB/s | 171 kB 00:01 2024-03-25T16:39:35.985 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:39:35.993 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:39:35.993 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:39:36.052 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:39:36.052 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:39:36.199 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:39:36.309 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:36.730 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:36.998 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:39:36.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:36.998 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:39:36.998 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:36.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:36.998 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:39:37.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:39:37.814 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:39:37.969 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:39:37.969 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:37.969 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T16:39:37.969 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:37.969 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:39:37.969 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:39:37.969 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-25T16:39:37.969 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:39:37.970 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:37.970 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:39:37.970 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:39:37.970 INFO:teuthology.orchestra.run.smithi102.stdout:Install 2 Packages 2024-03-25T16:39:37.970 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:39:37.970 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 91 M 2024-03-25T16:39:37.970 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 498 M 2024-03-25T16:39:37.971 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:39:40.095 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 302 kB/s | 7.4 MB 00:25 2024-03-25T16:39:40.098 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:40.098 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.5 MB/s | 37 MB 00:25 2024-03-25T16:39:40.140 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:39:40.165 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:39:40.166 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:39:40.525 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:39:40.525 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:39:41.250 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:39:41.327 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:39:41.358 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:39:41.459 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:39:41.503 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:39:41.675 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:39:41.702 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:39:41.786 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:39:42.438 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:39:42.564 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:39:43.801 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:39:44.827 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:39:44.850 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:39:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:39:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:39:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:39:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:39:47.352 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:39:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:39:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:39:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:39:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:39:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:39:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:39:47.353 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:39:47.765 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:39:47.766 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:39:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:39:47.767 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:47.767 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:39:48.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:39:48.820 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:06:39 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:39:48.970 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:39:48.971 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2024-03-25T16:39:48.972 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:48.973 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.0 M 2024-03-25T16:39:48.973 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-03-25T16:39:48.973 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:39:49.108 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 268 kB/s | 24 kB 00:00 2024-03-25T16:39:49.133 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 155 kB/s | 18 kB 00:00 2024-03-25T16:39:49.208 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-25T16:39:49.233 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:39:49.267 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:39:49.292 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 733 kB/s | 43 kB 00:00 2024-03-25T16:39:49.551 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 114 kB/s | 32 kB 00:00 2024-03-25T16:39:49.576 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-03-25T16:39:49.693 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 77 kB/s | 52 kB 00:00 2024-03-25T16:39:49.743 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.0 MB/s | 203 kB 00:00 2024-03-25T16:39:49.852 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-25T16:39:49.944 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-03-25T16:39:50.002 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 191 kB/s | 59 kB 00:00 2024-03-25T16:39:50.006 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:50.006 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-25T16:39:50.007 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:39:50.019 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:39:50.019 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:39:50.111 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:39:50.111 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:39:50.615 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:39:50.806 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:39:50.853 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:39:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:39:50.954 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:39:50.995 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:39:51.088 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:39:51.189 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:39:51.238 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:39:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:39:51.994 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:39:52.027 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:39:52.075 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:39:52.097 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:39:52.884 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:39:52.884 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:39:52.885 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:39:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:39:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:53.263 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:39:53.263 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:39:53.264 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:53.265 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:39:53.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:39:54.095 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:06:45 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:39:54.246 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:39:54.247 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T16:39:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:39:54.247 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T16:39:54.247 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:39:54.247 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:39:54.247 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-03-25T16:39:54.248 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:54.249 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.8 M 2024-03-25T16:39:54.249 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 33 M 2024-03-25T16:39:54.249 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:39:54.770 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.6 MB/s | 248 kB 00:00 2024-03-25T16:39:55.012 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.5 MB/s | 548 kB 00:00 2024-03-25T16:39:55.187 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-25T16:39:55.212 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 289 kB/s | 58 kB 00:00 2024-03-25T16:39:55.287 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 511 kB/s | 51 kB 00:00 2024-03-25T16:39:55.671 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 139 kB/s | 138 kB 00:00 2024-03-25T16:39:55.673 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:55.674 INFO:teuthology.orchestra.run.smithi146.stdout:Total 4.8 MB/s | 6.8 MB 00:01 2024-03-25T16:39:55.687 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:39:55.698 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:39:55.698 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:39:55.791 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:39:55.792 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:39:56.089 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:39:56.244 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:39:56.297 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:39:56.828 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:39:56.909 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:39:56.970 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:39:56.993 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:39:57.767 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:39:57.767 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:39:57.767 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:39:57.767 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:39:57.767 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:39:57.768 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:39:58.070 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:39:58.070 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:58.070 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:39:58.070 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:39:58.070 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:39:58.070 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:39:58.071 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:39:58.071 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:39:58.071 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:39:58.071 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:58.071 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:39:58.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse 2024-03-25T16:39:58.906 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:06:49 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:39:59.057 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:39:59.057 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:39:59.057 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:39:59.057 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:39:59.057 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:39:59.057 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:39:59.058 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:39:59.058 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:39:59.058 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:59.058 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:39:59.058 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:39:59.058 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-03-25T16:39:59.058 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:39:59.058 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 964 k 2024-03-25T16:39:59.059 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.8 M 2024-03-25T16:39:59.059 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:39:59.164 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 749 kB/s | 79 kB 00:00 2024-03-25T16:40:03.047 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 222 kB/s | 885 kB 00:03 2024-03-25T16:40:03.048 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:03.049 INFO:teuthology.orchestra.run.smithi146.stdout:Total 241 kB/s | 964 kB 00:03 2024-03-25T16:40:03.049 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:40:03.072 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:40:03.072 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:40:03.148 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:40:03.148 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:40:03.334 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:40:03.474 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:40:03.537 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:40:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:40:04.255 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:40:04.534 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:40:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:04.535 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:40:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:40:04.535 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:04.535 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:04.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-volume 2024-03-25T16:40:05.434 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:06:56 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:05.585 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:40:05.586 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:05.587 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-03-25T16:40:05.587 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:05.587 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 332 k 2024-03-25T16:40:05.587 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.5 M 2024-03-25T16:40:05.587 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:40:05.692 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 654 kB/s | 69 kB 00:00 2024-03-25T16:40:06.934 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 196 kB/s | 264 kB 00:01 2024-03-25T16:40:06.935 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:06.936 INFO:teuthology.orchestra.run.smithi146.stdout:Total 246 kB/s | 332 kB 00:01 2024-03-25T16:40:06.936 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:40:06.946 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:40:06.946 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:40:06.986 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:40:06.986 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:40:07.139 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:40:07.316 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:40:07.370 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:40:07.385 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:40:07.385 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:40:07.385 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:40:07.385 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:08.140 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:40:08.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:40:08.478 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:08.478 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:40:08.478 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:40:08.478 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:40:08.478 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:08.478 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:08.808 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel 2024-03-25T16:40:09.404 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:00 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:09.554 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:09.555 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:09.555 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T16:40:09.555 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:09.555 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:40:09.555 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:40:09.555 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:09.555 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:40:09.556 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:09.556 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T16:40:09.556 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:09.556 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 127 k 2024-03-25T16:40:09.556 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 456 k 2024-03-25T16:40:09.556 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:40:10.839 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 99 kB/s | 127 kB 00:01 2024-03-25T16:40:10.840 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:10.840 INFO:teuthology.orchestra.run.smithi146.stdout:Total 99 kB/s | 127 kB 00:01 2024-03-25T16:40:10.840 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:40:10.847 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:40:10.847 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:40:10.908 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:40:10.908 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:40:11.068 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:40:11.171 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:11.596 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:11.901 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:11.901 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:11.901 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:40:11.901 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:11.901 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:11.901 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:12.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2 2024-03-25T16:40:12.713 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:03 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:12.812 INFO:teuthology.orchestra.run.smithi146.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:12.866 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:12.867 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:40:12.867 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:12.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs-devel 2024-03-25T16:40:13.517 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:04 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:13.667 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:13.668 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 31 k 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 140 k 2024-03-25T16:40:13.669 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:40:14.325 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 47 kB/s | 31 kB 00:00 2024-03-25T16:40:14.326 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:14.326 INFO:teuthology.orchestra.run.smithi146.stdout:Total 47 kB/s | 31 kB 00:00 2024-03-25T16:40:14.326 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:40:14.330 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:40:14.330 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:40:14.358 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:40:14.359 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:40:14.492 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:40:14.624 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:14.999 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:15.266 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:15.266 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:15.266 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:40:15.267 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:15.267 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:15.267 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:15.436 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2 2024-03-25T16:40:16.031 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:07 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:16.134 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:16.189 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:16.190 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:40:16.190 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:16.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1 2024-03-25T16:40:16.832 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:07 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:16.934 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:16.989 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:16.990 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:40:16.990 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:17.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados 2024-03-25T16:40:17.605 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:08 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:17.702 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:17.753 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:17.754 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:40:17.754 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:17.807 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw 2024-03-25T16:40:18.366 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:09 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:18.461 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:18.512 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:18.513 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:40:18.513 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:18.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs 2024-03-25T16:40:19.126 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:10 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:19.224 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:19.276 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:19.277 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:40:19.277 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:19.329 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd 2024-03-25T16:40:19.894 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:10 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:19.990 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:40:20.046 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:20.046 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T16:40:20.046 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:20.095 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse 2024-03-25T16:40:20.666 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:11 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:20.815 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:20.816 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T16:40:20.817 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:20.817 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 85 k 2024-03-25T16:40:20.817 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 231 k 2024-03-25T16:40:20.817 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:40:21.630 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 105 kB/s | 85 kB 00:00 2024-03-25T16:40:21.631 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:21.631 INFO:teuthology.orchestra.run.smithi146.stdout:Total 105 kB/s | 85 kB 00:00 2024-03-25T16:40:21.631 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:40:21.639 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:40:21.639 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:40:21.699 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:40:21.735 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:40:21.837 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:40:21.945 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:22.375 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:22.664 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:22.664 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:22.664 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:40:22.665 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:22.665 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:22.665 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:22.873 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2024-03-25T16:40:23.453 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:14 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:23.606 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:23.607 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:23.607 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T16:40:23.607 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:23.607 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:40:23.607 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:40:23.607 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:23.607 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:40:23.607 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:23.608 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T16:40:23.608 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:23.608 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.1 M 2024-03-25T16:40:23.608 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 13 M 2024-03-25T16:40:23.608 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:40:35.499 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 266 kB/s | 3.1 MB 00:11 2024-03-25T16:40:35.500 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:35.500 INFO:teuthology.orchestra.run.smithi146.stdout:Total 266 kB/s | 3.1 MB 00:11 2024-03-25T16:40:35.500 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:40:35.510 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:40:35.510 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:40:35.587 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:40:35.587 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:40:36.023 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:40:36.082 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:36.106 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:36.106 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:40:36.106 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:40:36.106 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:40:36.107 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:40:36.107 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:37.240 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:37.240 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:37.240 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:40:37.240 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:37.241 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:37.241 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:37.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd 2024-03-25T16:40:38.071 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:29 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:38.221 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:38.221 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:38.221 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 171 k 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 494 k 2024-03-25T16:40:38.222 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:40:39.759 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 112 kB/s | 171 kB 00:01 2024-03-25T16:40:39.759 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:40:39.760 INFO:teuthology.orchestra.run.smithi146.stdout:Total 111 kB/s | 171 kB 00:01 2024-03-25T16:40:39.760 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:40:39.774 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:40:39.774 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:40:39.833 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:40:39.834 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:40:40.012 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:40:40.116 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:40.556 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:40.863 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:40:40.864 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:40.864 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:40:40.864 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:40:40.864 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:40.864 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:40:41.102 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:40:41.676 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:07:32 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:40:41.830 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:40:41.831 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T16:40:41.832 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:41.832 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:40:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:40:41.832 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-25T16:40:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:40:41.832 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:41.833 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:40:41.833 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:40:41.833 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-03-25T16:40:41.833 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:40:41.833 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 91 M 2024-03-25T16:40:41.833 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 498 M 2024-03-25T16:40:41.834 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T16:41:08.554 INFO:teuthology.orchestra.run.smithi102.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 254 kB/s | 22 MB 01:30 2024-03-25T16:42:18.200 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 239 kB/s | 22 MB 01:36 2024-03-25T16:43:49.161 INFO:teuthology.orchestra.run.smithi102.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 278 kB/s | 68 MB 04:11 2024-03-25T16:43:49.162 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:49.163 INFO:teuthology.orchestra.run.smithi102.stdout:Total 369 kB/s | 91 MB 04:11 2024-03-25T16:43:49.163 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T16:43:49.167 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T16:43:49.167 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T16:43:49.745 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T16:43:49.746 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T16:43:52.818 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T16:43:59.868 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:43:59.935 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:44:03.317 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:44:03.317 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:44:03.574 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:44:03.574 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:44:03.575 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T16:44:03.575 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:03.575 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:03.575 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:44:03.575 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T16:44:03.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:44:04.383 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:10:57 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T16:44:04.531 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T16:44:04.531 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:44:04.531 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T16:44:04.531 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 416 M 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 1.7 G 2024-03-25T16:44:04.532 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T16:45:42.322 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 232 kB/s | 68 MB 05:00 2024-03-25T16:45:42.323 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:45:42.324 INFO:teuthology.orchestra.run.smithi146.stdout:Total 309 kB/s | 91 MB 05:00 2024-03-25T16:45:42.324 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T16:45:42.328 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T16:45:42.328 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T16:45:42.908 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T16:45:42.908 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T16:45:46.011 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T16:45:53.099 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:45:53.239 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:45:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:45:56.492 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:45:56.773 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:45:56.773 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:45:56.773 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T16:45:56.773 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:56.773 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:45:56.773 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:45:56.773 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T16:45:57.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:45:57.592 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:12:48 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T16:45:57.742 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 416 M 2024-03-25T16:45:57.743 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.7 G 2024-03-25T16:45:57.744 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T17:00:16.085 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 439 kB/s | 416 MB 16:11 2024-03-25T17:00:16.085 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:00:16.085 INFO:teuthology.orchestra.run.smithi102.stdout:Total 439 kB/s | 416 MB 16:11 2024-03-25T17:00:16.085 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T17:00:16.088 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T17:00:16.088 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T17:00:18.559 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T17:00:18.560 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T17:01:00.689 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T17:01:00.795 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:01:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:01:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:01:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:01:08.207 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T17:01:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:01:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:01:08.208 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T17:01:08.458 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:01:09.020 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:28:02 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T17:01:09.168 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================== 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T17:01:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:01:09.170 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 1.4 M 2024-03-25T17:01:09.170 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 6.4 M 2024-03-25T17:01:09.170 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T17:01:12.291 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 468 kB/s | 1.4 MB 00:03 2024-03-25T17:01:12.292 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:01:12.292 INFO:teuthology.orchestra.run.smithi102.stdout:Total 468 kB/s | 1.4 MB 00:03 2024-03-25T17:01:12.293 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T17:01:12.297 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T17:01:12.297 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T17:01:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T17:01:12.340 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T17:01:12.613 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T17:01:12.714 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:01:13.117 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:01:13.377 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:01:13.377 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:01:13.377 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T17:01:13.377 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:01:13.377 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:01:13.377 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T17:01:13.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:01:14.120 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:28:07 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T17:01:14.269 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T17:01:14.269 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 176 M 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 753 M 2024-03-25T17:01:14.270 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T17:13:08.342 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 252 kB/s | 176 MB 11:54 2024-03-25T17:13:08.342 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:08.342 INFO:teuthology.orchestra.run.smithi102.stdout:Total 252 kB/s | 176 MB 11:54 2024-03-25T17:13:08.342 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T17:13:08.345 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T17:13:08.345 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T17:13:09.405 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T17:13:09.406 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T17:13:27.614 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T17:13:28.105 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:13:30.769 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:13:31.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:13:31.032 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:13:31.032 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T17:13:31.032 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:31.032 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:13:31.032 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T17:13:31.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:13:31.783 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:40:24 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T17:13:31.931 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 672 M 2024-03-25T17:13:31.932 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 3.1 G 2024-03-25T17:13:31.933 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T17:14:33.612 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 248 kB/s | 416 MB 28:35 2024-03-25T17:14:33.613 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:33.613 INFO:teuthology.orchestra.run.smithi146.stdout:Total 248 kB/s | 416 MB 28:35 2024-03-25T17:14:33.613 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T17:14:33.616 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T17:14:33.617 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T17:14:36.090 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T17:14:36.090 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T17:15:18.486 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T17:15:18.573 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:25.362 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:25.630 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:15:25.631 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:15:25.631 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T17:15:25.631 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:25.631 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:15:25.631 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T17:15:25.877 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:15:26.448 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:42:17 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T17:15:26.598 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T17:15:26.599 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T17:15:26.599 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T17:15:26.599 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 1.4 M 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 6.4 M 2024-03-25T17:15:26.600 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T17:15:31.942 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 274 kB/s | 1.4 MB 00:05 2024-03-25T17:15:31.943 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:31.943 INFO:teuthology.orchestra.run.smithi146.stdout:Total 273 kB/s | 1.4 MB 00:05 2024-03-25T17:15:31.943 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T17:15:31.947 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T17:15:31.948 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T17:15:31.993 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T17:15:31.994 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T17:15:32.244 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T17:15:32.357 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:15:32.769 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:15:33.054 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:15:33.054 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:15:33.054 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T17:15:33.055 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:33.055 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:15:33.055 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T17:15:33.243 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:15:33.811 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:42:24 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T17:15:33.961 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 176 M 2024-03-25T17:15:33.962 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 753 M 2024-03-25T17:15:33.963 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T17:24:01.292 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 355 kB/s | 176 MB 08:27 2024-03-25T17:24:01.335 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:24:01.335 INFO:teuthology.orchestra.run.smithi146.stdout:Total 355 kB/s | 176 MB 08:27 2024-03-25T17:24:01.336 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T17:24:01.336 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T17:24:01.336 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T17:24:02.356 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T17:24:02.357 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T17:24:20.421 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T17:24:20.730 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:24:25.161 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:24:25.451 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:24:25.451 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:24:25.452 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T17:24:25.452 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:24:25.452 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:24:25.452 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T17:24:25.706 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:24:26.270 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:51:17 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T17:24:26.425 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T17:24:26.426 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:24:26.427 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 672 M 2024-03-25T17:24:26.427 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 3.1 G 2024-03-25T17:24:26.427 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T17:46:47.728 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 345 kB/s | 672 MB 33:15 2024-03-25T17:46:47.751 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:46:47.751 INFO:teuthology.orchestra.run.smithi102.stdout:Total 345 kB/s | 672 MB 33:15 2024-03-25T17:46:47.751 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T17:46:47.751 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T17:46:47.751 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T17:46:51.706 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T17:46:51.706 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T17:48:02.628 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T17:48:02.813 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:48:12.296 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:48:12.558 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:48:12.558 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:48:12.558 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T17:48:12.558 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:12.558 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:48:12.559 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T17:48:12.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:48:13.410 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:15:06 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T17:48:13.506 INFO:teuthology.orchestra.run.smithi102.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:48:13.558 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T17:48:13.559 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-03-25T17:48:13.559 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T17:48:13.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:48:14.193 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:15:07 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T17:48:14.342 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:48:14.343 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T17:48:14.344 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:48:14.344 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 20 M 2024-03-25T17:48:14.344 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 103 M 2024-03-25T17:48:14.344 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T17:49:22.252 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 297 kB/s | 20 MB 01:07 2024-03-25T17:49:22.253 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:22.253 INFO:teuthology.orchestra.run.smithi102.stdout:Total 297 kB/s | 20 MB 01:07 2024-03-25T17:49:22.253 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T17:49:22.257 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T17:49:22.257 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T17:49:22.408 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T17:49:22.408 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T17:49:24.674 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T17:49:24.776 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:25.725 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:25.990 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:25.990 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:49:25.990 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T17:49:25.990 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:25.990 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:49:25.990 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T17:49:26.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:49:26.731 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:16:19 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T17:49:26.879 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T17:49:26.880 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:49:26.880 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T17:49:26.880 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 48 M 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 173 M 2024-03-25T17:49:26.881 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T17:51:34.983 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 381 kB/s | 48 MB 02:08 2024-03-25T17:51:34.984 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:34.984 INFO:teuthology.orchestra.run.smithi102.stdout:Total 381 kB/s | 48 MB 02:08 2024-03-25T17:51:34.984 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T17:51:34.987 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T17:51:34.987 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T17:51:35.299 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T17:51:35.299 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T17:51:40.017 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T17:51:40.114 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:41.431 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:41.697 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:41.697 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:51:41.697 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T17:51:41.697 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:41.697 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:51:41.697 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T17:51:41.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:51:42.442 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:18:35 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T17:51:42.592 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T17:51:42.592 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 99 M 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 449 M 2024-03-25T17:51:42.593 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T17:54:51.790 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 533 kB/s | 99 MB 03:09 2024-03-25T17:54:51.791 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:51.791 INFO:teuthology.orchestra.run.smithi102.stdout:Total 533 kB/s | 99 MB 03:09 2024-03-25T17:54:51.791 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T17:54:51.793 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T17:54:51.793 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T17:54:52.400 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T17:54:52.400 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T17:55:02.679 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T17:55:02.778 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:05.672 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:05.938 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:05.938 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:55:05.938 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T17:55:05.938 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:05.938 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:55:05.938 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T17:55:06.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:55:06.686 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:21:59 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T17:55:06.836 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T17:55:06.837 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T17:55:06.837 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T17:55:06.837 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 287 M 2024-03-25T17:55:06.837 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 1.2 G 2024-03-25T17:55:06.837 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T17:57:23.024 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 348 kB/s | 672 MB 32:56 2024-03-25T17:57:23.025 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:23.025 INFO:teuthology.orchestra.run.smithi146.stdout:Total 348 kB/s | 672 MB 32:56 2024-03-25T17:57:23.025 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T17:57:23.029 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T17:57:23.029 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T17:57:26.982 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T17:57:26.982 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T17:58:37.819 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T17:58:37.922 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:46.566 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:58:46.822 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T17:58:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:46.822 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:58:46.822 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T17:58:47.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:58:47.714 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:25:38 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T17:58:47.811 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:58:47.864 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T17:58:47.864 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-03-25T17:58:47.865 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T17:58:47.931 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:58:48.500 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:25:39 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T17:58:48.648 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T17:58:48.649 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:58:48.649 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T17:58:48.649 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:58:48.649 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T17:58:48.649 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:58:48.650 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:58:48.650 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T17:58:48.650 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T17:58:48.650 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T17:58:48.650 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T17:58:48.650 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 20 M 2024-03-25T17:58:48.650 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 103 M 2024-03-25T17:58:48.650 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:00:03.551 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 269 kB/s | 20 MB 01:14 2024-03-25T18:00:03.551 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:03.552 INFO:teuthology.orchestra.run.smithi146.stdout:Total 269 kB/s | 20 MB 01:14 2024-03-25T18:00:03.552 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:00:03.554 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:00:03.554 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:00:03.702 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:00:03.702 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:00:05.927 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:00:06.097 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:07.046 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:07.308 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:07.308 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:00:07.309 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:00:07.309 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:07.309 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:00:07.309 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:00:07.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:00:08.051 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:26:59 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:00:08.200 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 48 M 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 173 M 2024-03-25T18:00:08.201 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:03:04.821 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 277 kB/s | 48 MB 02:56 2024-03-25T18:03:04.821 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:04.821 INFO:teuthology.orchestra.run.smithi146.stdout:Total 277 kB/s | 48 MB 02:56 2024-03-25T18:03:04.821 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:03:04.824 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:03:04.824 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:03:05.136 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:03:05.136 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:03:09.889 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:03:09.984 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:11.250 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:03:11.523 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:03:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:11.523 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:03:11.523 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:03:11.743 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:03:12.332 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:30:03 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:03:12.494 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:03:12.495 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:03:12.496 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:03:12.496 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 99 M 2024-03-25T18:03:12.496 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 449 M 2024-03-25T18:03:12.496 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:09:15.364 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 278 kB/s | 99 MB 06:02 2024-03-25T18:09:15.365 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:15.365 INFO:teuthology.orchestra.run.smithi146.stdout:Total 278 kB/s | 99 MB 06:02 2024-03-25T18:09:15.365 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:09:15.367 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:09:15.368 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:09:15.975 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:09:15.975 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:09:26.208 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:09:26.286 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:09:29.103 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:09:29.362 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:09:29.363 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:09:29.363 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:09:29.363 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:29.363 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:09:29.363 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:09:29.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:09:30.138 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:36:21 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:09:30.287 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 287 M 2024-03-25T18:09:30.288 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.2 G 2024-03-25T18:09:30.289 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:10:08.301 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 326 kB/s | 287 MB 15:01 2024-03-25T18:10:08.301 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:08.301 INFO:teuthology.orchestra.run.smithi102.stdout:Total 326 kB/s | 287 MB 15:01 2024-03-25T18:10:08.302 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:10:08.306 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:10:08.306 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:10:10.017 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:10:10.018 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:10:39.380 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:10:39.472 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:46.946 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:47.217 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:47.217 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:10:47.217 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:10:47.217 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:47.217 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:10:47.217 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:10:47.427 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:10:47.995 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:37:40 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:10:48.142 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:10:48.143 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:10:48.143 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:10:48.143 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:10:48.143 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:10:48.143 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:10:48.144 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:10:48.144 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:10:48.144 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:10:48.144 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:10:48.144 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:10:48.144 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 5.9 M 2024-03-25T18:10:48.144 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 28 M 2024-03-25T18:10:48.144 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:11:06.366 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 334 kB/s | 5.9 MB 00:18 2024-03-25T18:11:06.367 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:06.367 INFO:teuthology.orchestra.run.smithi102.stdout:Total 334 kB/s | 5.9 MB 00:18 2024-03-25T18:11:06.367 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:11:06.371 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:11:06.371 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:11:06.442 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:11:06.442 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:11:07.178 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:11:07.289 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:11:07.901 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:11:08.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:38:01 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 109 k 2024-03-25T18:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 440 k 2024-03-25T18:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:11:09.802 INFO:teuthology.orchestra.run.smithi102.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 150 kB/s | 109 kB 00:00 2024-03-25T18:11:09.803 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:09.803 INFO:teuthology.orchestra.run.smithi102.stdout:Total 150 kB/s | 109 kB 00:00 2024-03-25T18:11:09.803 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:11:09.807 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:11:09.807 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:11:09.843 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:11:09.844 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:11:09.964 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:11:10.068 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:10.474 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:10.742 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:10.743 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:10.743 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:11:10.743 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:10.743 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:10.743 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:11:10.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:11:11.492 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:38:04 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:11:11.640 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:11:11.641 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:11.642 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 5.4 M 2024-03-25T18:11:11.642 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 25 M 2024-03-25T18:11:11.642 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:11:27.737 INFO:teuthology.orchestra.run.smithi102.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 341 kB/s | 5.4 MB 00:16 2024-03-25T18:11:27.738 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:27.738 INFO:teuthology.orchestra.run.smithi102.stdout:Total 340 kB/s | 5.4 MB 00:16 2024-03-25T18:11:27.739 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:11:27.742 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:11:27.743 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:11:27.810 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:11:27.810 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:11:28.482 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:11:28.587 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:11:29.154 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:11:29.424 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:11:29.424 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:29.425 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:11:29.425 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:29.425 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:29.425 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:11:29.627 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librados2-debuginfo 2024-03-25T18:11:30.188 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:38:23 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:11:30.337 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 44 M 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 253 M 2024-03-25T18:11:30.338 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:13:12.796 INFO:teuthology.orchestra.run.smithi102.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 443 kB/s | 44 MB 01:42 2024-03-25T18:13:12.797 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:12.798 INFO:teuthology.orchestra.run.smithi102.stdout:Total 443 kB/s | 44 MB 01:42 2024-03-25T18:13:12.798 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:13:12.801 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:13:12.801 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:13:13.100 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:13:13.100 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:13:18.161 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:13:18.261 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:13:20.102 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:13:20.376 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:13:20.376 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:13:20.376 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:13:20.376 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:20.376 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:13:20.376 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:13:20.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:13:21.156 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:40:14 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:13:21.307 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:13:21.308 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:13:21.308 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:13:21.308 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 52 M 2024-03-25T18:13:21.308 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 308 M 2024-03-25T18:13:21.308 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:15:34.961 INFO:teuthology.orchestra.run.smithi102.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 397 kB/s | 52 MB 02:13 2024-03-25T18:15:34.961 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:34.961 INFO:teuthology.orchestra.run.smithi102.stdout:Total 397 kB/s | 52 MB 02:13 2024-03-25T18:15:34.961 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:15:34.964 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:15:34.964 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:15:35.310 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:15:35.310 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:15:41.307 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:15:41.475 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:43.661 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:43.931 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:43.931 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:43.931 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:15:43.931 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:43.931 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:43.931 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:15:44.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:15:44.711 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:42:37 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:15:44.860 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:15:44.861 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:44.862 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 615 k 2024-03-25T18:15:44.862 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 2.1 M 2024-03-25T18:15:44.862 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:15:46.962 INFO:teuthology.orchestra.run.smithi102.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 293 kB/s | 615 kB 00:02 2024-03-25T18:15:46.963 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:46.963 INFO:teuthology.orchestra.run.smithi102.stdout:Total 293 kB/s | 615 kB 00:02 2024-03-25T18:15:46.964 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:15:46.967 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:15:46.967 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:15:47.008 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:15:47.009 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:15:47.181 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:15:47.288 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:47.701 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:47.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:47.974 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:47.974 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:15:47.974 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:47.974 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:47.974 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:15:48.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:15:48.721 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:42:41 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:15:48.872 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:15:48.873 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:48.873 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:15:48.873 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 771 k 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 2.7 M 2024-03-25T18:15:48.874 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:15:50.999 INFO:teuthology.orchestra.run.smithi102.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 363 kB/s | 771 kB 00:02 2024-03-25T18:15:50.999 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:51.000 INFO:teuthology.orchestra.run.smithi102.stdout:Total 363 kB/s | 771 kB 00:02 2024-03-25T18:15:51.000 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:15:51.004 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:15:51.004 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:15:51.045 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:15:51.046 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:15:51.227 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:15:51.323 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:51.692 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:51.965 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:51.965 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:51.965 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:15:51.966 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:51.966 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:51.966 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:15:52.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:15:52.754 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:42:45 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:15:52.903 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 1.1 M 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 4.3 M 2024-03-25T18:15:52.904 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:15:55.696 INFO:teuthology.orchestra.run.smithi102.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 412 kB/s | 1.1 MB 00:02 2024-03-25T18:15:55.696 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:55.697 INFO:teuthology.orchestra.run.smithi102.stdout:Total 412 kB/s | 1.1 MB 00:02 2024-03-25T18:15:55.697 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:15:55.701 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:15:55.701 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:15:55.745 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:15:55.745 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:15:55.982 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:15:56.096 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:56.494 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:56.771 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:56.771 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:56.771 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:15:56.771 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:56.771 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:56.772 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:15:56.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:15:57.559 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:42:50 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:15:57.713 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:15:57.713 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:57.713 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:15:57.713 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:57.713 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:15:57.713 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:15:57.714 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:57.714 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:15:57.714 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:15:57.714 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:15:57.714 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:15:57.714 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 337 k 2024-03-25T18:15:57.715 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 1.0 M 2024-03-25T18:15:57.715 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:15:59.090 INFO:teuthology.orchestra.run.smithi102.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 245 kB/s | 337 kB 00:01 2024-03-25T18:15:59.091 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:59.091 INFO:teuthology.orchestra.run.smithi102.stdout:Total 245 kB/s | 337 kB 00:01 2024-03-25T18:15:59.091 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:15:59.095 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:15:59.095 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:15:59.135 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:15:59.135 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:15:59.276 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:15:59.381 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:59.736 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:16:00.006 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:16:00.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:16:00.006 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:16:00.006 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:00.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:16:00.007 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:16:00.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:16:00.785 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:42:53 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:16:00.934 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:16:00.935 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:16:00.935 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:16:00.935 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:16:00.935 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:16:00.935 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:16:00.936 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:16:00.936 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:16:00.936 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:16:00.936 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:16:00.936 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:16:00.936 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 509 k 2024-03-25T18:16:00.936 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 2.5 M 2024-03-25T18:16:00.936 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:16:02.261 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 384 kB/s | 509 kB 00:01 2024-03-25T18:16:02.261 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:02.262 INFO:teuthology.orchestra.run.smithi102.stdout:Total 384 kB/s | 509 kB 00:01 2024-03-25T18:16:02.262 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:16:02.265 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:16:02.266 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:16:02.305 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:16:02.306 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:16:02.465 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:16:02.577 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:02.966 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:03.239 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:03.239 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:16:03.239 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:16:03.239 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:03.239 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:16:03.239 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:16:03.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:16:04.009 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:42:57 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:16:04.159 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:16:04.160 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:16:04.160 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:16:04.160 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:16:04.160 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:16:04.160 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:16:04.160 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:16:04.160 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:16:04.161 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:16:04.161 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:16:04.161 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:16:04.161 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 52 M 2024-03-25T18:16:04.161 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 298 M 2024-03-25T18:16:04.161 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:18:11.418 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 564 kB/s | 287 MB 08:41 2024-03-25T18:18:11.418 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:11.418 INFO:teuthology.orchestra.run.smithi146.stdout:Total 564 kB/s | 287 MB 08:41 2024-03-25T18:18:11.418 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:18:11.421 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:18:11.421 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:18:13.125 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:18:13.126 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:18:24.923 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 375 kB/s | 52 MB 02:20 2024-03-25T18:18:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:Total 375 kB/s | 52 MB 02:20 2024-03-25T18:18:24.924 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:18:24.927 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:18:24.928 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:18:25.260 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:18:25.260 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:18:31.182 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:18:31.405 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:33.503 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:18:33.776 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:18:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:33.776 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:18:33.776 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:18:33.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:18:34.542 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 1:45:27 ago on Mon 25 Mar 2024 04:33:07 PM UTC. 2024-03-25T18:18:34.690 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-25T18:18:34.691 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:18:34.691 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repo Size 2024-03-25T18:18:34.691 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:18:34.691 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-25T18:18:34.691 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:18:34.691 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:18:34.692 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-25T18:18:34.692 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-25T18:18:34.692 INFO:teuthology.orchestra.run.smithi102.stdout:Install 1 Package 2024-03-25T18:18:34.692 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:18:34.692 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 1.3 M 2024-03-25T18:18:34.692 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 5.5 M 2024-03-25T18:18:34.693 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-25T18:18:37.826 INFO:teuthology.orchestra.run.smithi102.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 409 kB/s | 1.3 MB 00:03 2024-03-25T18:18:37.826 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:37.827 INFO:teuthology.orchestra.run.smithi102.stdout:Total 409 kB/s | 1.3 MB 00:03 2024-03-25T18:18:37.827 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-25T18:18:37.830 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-25T18:18:37.831 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-25T18:18:37.875 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-25T18:18:37.875 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-25T18:18:38.113 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-25T18:18:38.216 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:38.658 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:38.925 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:38.925 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:18:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-25T18:18:38.925 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:38.925 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:18:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-25T18:18:39.107 DEBUG:teuthology.parallel:result is None 2024-03-25T18:18:42.493 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:18:42.595 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:49.929 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:18:50.207 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:18:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:50.207 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:18:50.208 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:18:50.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:18:51.010 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:45:42 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:18:51.158 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:18:51.159 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:18:51.159 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:18:51.159 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:18:51.159 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:18:51.159 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:18:51.159 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:18:51.160 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:18:51.160 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:18:51.160 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:18:51.160 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:18:51.160 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 5.9 M 2024-03-25T18:18:51.160 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 28 M 2024-03-25T18:18:51.160 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:19:07.655 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 369 kB/s | 5.9 MB 00:16 2024-03-25T18:19:07.656 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:07.656 INFO:teuthology.orchestra.run.smithi146.stdout:Total 369 kB/s | 5.9 MB 00:16 2024-03-25T18:19:07.657 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:19:07.660 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:19:07.661 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:19:07.731 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:19:07.731 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:19:08.465 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:19:08.554 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:09.130 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:09.404 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:19:09.405 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:09.405 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:19:09.405 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:09.405 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:09.405 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:19:09.623 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:19:10.189 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:46:01 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:19:10.340 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:19:10.341 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:10.342 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 109 k 2024-03-25T18:19:10.342 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 440 k 2024-03-25T18:19:10.342 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:19:11.249 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 120 kB/s | 109 kB 00:00 2024-03-25T18:19:11.250 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:11.251 INFO:teuthology.orchestra.run.smithi146.stdout:Total 119 kB/s | 109 kB 00:00 2024-03-25T18:19:11.251 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:19:11.254 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:19:11.254 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:19:11.290 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:19:11.290 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:19:11.412 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:19:11.507 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:11.870 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:12.144 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:19:12.144 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:12.144 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:19:12.144 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:12.144 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:12.144 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:19:12.355 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:19:12.923 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:46:03 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:19:13.078 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:13.079 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 5.4 M 2024-03-25T18:19:13.080 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 25 M 2024-03-25T18:19:13.080 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:19:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 562 kB/s | 5.4 MB 00:09 2024-03-25T18:19:22.835 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:Total 562 kB/s | 5.4 MB 00:09 2024-03-25T18:19:22.836 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:19:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:19:22.840 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:19:22.908 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:19:22.908 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:19:23.577 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:19:23.694 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:19:24.201 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:19:24.474 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:19:24.475 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:24.475 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:19:24.475 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:19:24.475 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:24.475 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:19:24.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2-debuginfo 2024-03-25T18:19:25.247 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:46:16 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:19:25.395 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:19:25.396 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:19:25.396 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:25.396 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:19:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:19:25.396 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:25.397 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:19:25.397 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:19:25.397 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:19:25.397 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:19:25.397 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 44 M 2024-03-25T18:19:25.397 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 253 M 2024-03-25T18:19:25.397 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:21:18.431 INFO:teuthology.orchestra.run.smithi146.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 401 kB/s | 44 MB 01:53 2024-03-25T18:21:18.432 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:18.432 INFO:teuthology.orchestra.run.smithi146.stdout:Total 401 kB/s | 44 MB 01:53 2024-03-25T18:21:18.432 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:21:18.435 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:21:18.435 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:21:18.735 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:21:18.735 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:21:23.809 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:21:23.888 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:21:25.636 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:21:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:21:25.907 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:21:25.908 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:21:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:25.908 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:21:25.908 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:21:26.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:21:26.685 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:48:17 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:21:26.836 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:21:26.837 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:21:26.837 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:21:26.837 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:21:26.837 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:21:26.837 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:21:26.837 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:21:26.837 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:21:26.838 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:21:26.838 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:21:26.838 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:21:26.838 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 52 M 2024-03-25T18:21:26.838 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 308 M 2024-03-25T18:21:26.838 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:22:56.154 INFO:teuthology.orchestra.run.smithi146.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 594 kB/s | 52 MB 01:29 2024-03-25T18:22:56.155 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:22:56.155 INFO:teuthology.orchestra.run.smithi146.stdout:Total 594 kB/s | 52 MB 01:29 2024-03-25T18:22:56.155 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:22:56.159 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:22:56.159 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:22:56.503 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:22:56.503 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:23:02.480 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:23:02.611 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:04.703 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:23:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:04.974 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:23:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:04.974 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:04.974 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:23:05.225 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:23:05.786 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:49:56 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:23:05.936 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:05.938 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 615 k 2024-03-25T18:23:05.938 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.1 M 2024-03-25T18:23:05.938 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:23:07.848 INFO:teuthology.orchestra.run.smithi146.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 322 kB/s | 615 kB 00:01 2024-03-25T18:23:07.849 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:07.849 INFO:teuthology.orchestra.run.smithi146.stdout:Total 322 kB/s | 615 kB 00:01 2024-03-25T18:23:07.849 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:23:07.853 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:23:07.853 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:23:07.896 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:23:07.896 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:23:08.067 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:23:08.172 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:08.582 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:08.847 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:23:08.847 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:08.847 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:23:08.847 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:08.847 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:08.847 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:23:09.055 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:23:09.620 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:50:00 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:23:09.772 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:23:09.773 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:09.774 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 771 k 2024-03-25T18:23:09.774 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.7 M 2024-03-25T18:23:09.774 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:23:11.692 INFO:teuthology.orchestra.run.smithi146.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 402 kB/s | 771 kB 00:01 2024-03-25T18:23:11.693 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:11.694 INFO:teuthology.orchestra.run.smithi146.stdout:Total 401 kB/s | 771 kB 00:01 2024-03-25T18:23:11.694 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:23:11.698 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:23:11.698 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:23:11.739 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:23:11.740 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:23:11.922 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:23:12.016 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:12.406 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:23:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:12.698 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:23:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:12.698 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:12.698 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:23:12.943 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:23:13.506 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:50:04 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:23:13.654 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:23:13.655 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:13.655 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:23:13.655 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:13.655 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:23:13.655 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:23:13.655 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:13.655 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:23:13.655 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:13.656 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:23:13.656 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:13.656 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 1.1 M 2024-03-25T18:23:13.656 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 4.3 M 2024-03-25T18:23:13.656 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:23:16.037 INFO:teuthology.orchestra.run.smithi146.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 483 kB/s | 1.1 MB 00:02 2024-03-25T18:23:16.037 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:16.038 INFO:teuthology.orchestra.run.smithi146.stdout:Total 483 kB/s | 1.1 MB 00:02 2024-03-25T18:23:16.038 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:23:16.042 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:23:16.042 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:23:16.085 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:23:16.085 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:23:16.309 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:23:16.420 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:16.831 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:17.096 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:23:17.096 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:17.097 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:17.097 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:23:17.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:23:17.876 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:50:08 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:23:18.026 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:23:18.027 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:23:18.027 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:18.027 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:23:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:23:18.027 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:18.028 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:23:18.028 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:18.028 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:23:18.028 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:18.028 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 337 k 2024-03-25T18:23:18.028 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.0 M 2024-03-25T18:23:18.028 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:23:19.048 INFO:teuthology.orchestra.run.smithi146.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 331 kB/s | 337 kB 00:01 2024-03-25T18:23:19.049 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:19.049 INFO:teuthology.orchestra.run.smithi146.stdout:Total 330 kB/s | 337 kB 00:01 2024-03-25T18:23:19.049 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:23:19.053 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:23:19.053 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:23:19.093 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:23:19.094 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:23:19.237 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:23:19.330 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:19.707 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:19.981 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:23:19.981 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:19.982 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:23:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:19.982 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:19.982 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:23:20.200 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:23:20.764 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:50:11 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:23:20.914 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:23:20.914 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:20.914 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:23:20.914 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:20.914 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:23:20.914 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:23:20.914 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:20.914 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:23:20.915 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:20.915 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:23:20.915 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:20.915 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 509 k 2024-03-25T18:23:20.915 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.5 M 2024-03-25T18:23:20.915 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:23:23.093 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 234 kB/s | 509 kB 00:02 2024-03-25T18:23:23.093 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:23.094 INFO:teuthology.orchestra.run.smithi146.stdout:Total 234 kB/s | 509 kB 00:02 2024-03-25T18:23:23.094 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:23:23.098 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:23:23.098 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:23:23.137 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:23:23.137 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:23:23.299 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:23:23.408 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:23.810 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:24.070 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:23:24.070 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:24.070 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:23:24.071 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:24.071 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:24.071 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:23:24.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:23:24.845 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:50:15 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:23:24.993 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:23:24.993 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 52 M 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 298 M 2024-03-25T18:23:24.994 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:24:53.543 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 596 kB/s | 52 MB 01:28 2024-03-25T18:24:53.543 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:53.543 INFO:teuthology.orchestra.run.smithi146.stdout:Total 596 kB/s | 52 MB 01:28 2024-03-25T18:24:53.544 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:24:53.547 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:24:53.547 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:24:53.882 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:24:53.882 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:24:59.809 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:24:59.915 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:02.189 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:25:02.189 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:02.190 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:25:02.190 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:02.190 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:02.190 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:25:02.399 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:25:02.967 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 1:51:53 ago on Mon 25 Mar 2024 04:33:09 PM UTC. 2024-03-25T18:25:03.117 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-25T18:25:03.118 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:25:03.118 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-03-25T18:25:03.118 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:25:03.118 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 1.3 M 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 5.5 M 2024-03-25T18:25:03.119 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-25T18:25:06.444 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 386 kB/s | 1.3 MB 00:03 2024-03-25T18:25:06.444 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:06.445 INFO:teuthology.orchestra.run.smithi146.stdout:Total 385 kB/s | 1.3 MB 00:03 2024-03-25T18:25:06.445 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-25T18:25:06.449 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-25T18:25:06.449 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-25T18:25:06.494 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-25T18:25:06.494 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-25T18:25:06.737 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-25T18:25:06.852 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:07.279 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:07.554 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:07.555 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-25T18:25:07.768 DEBUG:teuthology.parallel:result is None 2024-03-25T18:25:07.768 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:25:07.909 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:25:07.933 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:25:07.933 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:25:07.933 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:25:07.935 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:25:08.079 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:25:08.099 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:25:08.099 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:25:08.100 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:25:08.101 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:25:08.101 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:08.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:25:08.133 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:08.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:25:08.167 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:25:08.167 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:08.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:25:08.203 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:25:08.272 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:08.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:25:08.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:25:08.367 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:25:08.367 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:08.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:25:08.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:25:08.462 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:08.463 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:25:08.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:25:08.556 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:25:08.556 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:08.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:25:08.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:25:08.650 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:08.650 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:25:08.677 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:25:08.744 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:25:08.828 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:25:08.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /var/log/ceph 2024-03-25T18:25:08.830 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-03-25T18:25:08.856 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:25:08.856 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:25:08.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:25:08.923 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:25:08.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:25:08.963 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:25:09.018 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:25:09.018 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:25:09.018 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621280', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi102.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBAYvnjUY3nI+HIYO8Dn/SgNRNe9ZigdnjOdZ4vy2p9LJP7xZ4RUQ3G8wxeCTNr4a7yQkLUGc2eFGGr0tXpEf8=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDmwdMj0KKr+QmizJqJ9GC8PcQVl5kT7qN2x5x7cetOt//LXCqLl6za4k2jkJoSSdf8yvXr3HCWWda3EgwHDTI8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:25:09.019 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:25:09.029 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:25:09.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:25:09.084 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:25:09.111 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:09.111 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:25:09.163 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:25:09.163 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-25 16:39:33.094606872 +0000 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-25 16:32:03.648736110 +0000 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-25 16:32:03.648736110 +0000 2024-03-25T18:25:09.218 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-25 16:32:03.648736110 +0000 2024-03-25T18:25:09.219 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:25:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-25T18:25:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-25T18:25:09.283 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.00020856 s, 2.5 MB/s 2024-03-25T18:25:09.284 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:25:09.339 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-03-25T18:25:09.394 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:25:09.394 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:09.394 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:25:09.394 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:09.395 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:09.395 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-25 16:39:33.094606872 +0000 2024-03-25T18:25:09.395 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-25 16:32:04.002728727 +0000 2024-03-25T18:25:09.395 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-25 16:32:04.002728727 +0000 2024-03-25T18:25:09.395 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-25 16:32:04.002728727 +0000 2024-03-25T18:25:09.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:25:09.460 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-25T18:25:09.460 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-25T18:25:09.460 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.00020068 s, 2.6 MB/s 2024-03-25T18:25:09.461 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:25:09.516 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-25 16:39:33.094606872 +0000 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-25 16:32:04.349721490 +0000 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-25 16:32:04.349721490 +0000 2024-03-25T18:25:09.571 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-25 16:32:04.349721490 +0000 2024-03-25T18:25:09.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:25:09.636 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-25T18:25:09.636 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-25T18:25:09.636 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000169483 s, 3.0 MB/s 2024-03-25T18:25:09.637 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:25:09.693 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-25 16:39:33.095606851 +0000 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-25 16:32:04.679714607 +0000 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-25 16:32:04.679714607 +0000 2024-03-25T18:25:09.748 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-25 16:32:04.679714607 +0000 2024-03-25T18:25:09.748 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:25:09.813 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-25T18:25:09.813 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-25T18:25:09.813 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000186159 s, 2.8 MB/s 2024-03-25T18:25:09.814 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:25:09.869 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:09.870 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:09.870 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:25:09.887 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:25:09.887 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-03-25T18:25:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:25:09.943 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:09.944 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:25:09.944 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:09.944 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:09.944 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-25 16:40:36.344756465 +0000 2024-03-25T18:25:09.944 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-25 16:32:15.696267888 +0000 2024-03-25T18:25:09.944 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-25 16:32:15.696267888 +0000 2024-03-25T18:25:09.944 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-25 16:32:15.696267888 +0000 2024-03-25T18:25:09.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:25:10.012 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-25T18:25:10.012 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-25T18:25:10.012 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000225056 s, 2.3 MB/s 2024-03-25T18:25:10.014 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:25:10.071 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-25 16:40:36.344756465 +0000 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-25 16:32:16.053261736 +0000 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-25 16:32:16.053261736 +0000 2024-03-25T18:25:10.127 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-25 16:32:16.053261736 +0000 2024-03-25T18:25:10.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:25:10.194 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-25T18:25:10.195 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-25T18:25:10.195 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000249508 s, 2.1 MB/s 2024-03-25T18:25:10.196 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:25:10.253 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-25 16:40:36.344756465 +0000 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-25 16:32:16.384256032 +0000 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-25 16:32:16.384256032 +0000 2024-03-25T18:25:10.309 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-25 16:32:16.384256032 +0000 2024-03-25T18:25:10.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:25:10.376 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-25T18:25:10.376 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-25T18:25:10.376 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000189829 s, 2.7 MB/s 2024-03-25T18:25:10.377 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:25:10.433 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-03-25T18:25:10.488 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:25:10.488 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:25:10.489 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T18:25:10.489 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:25:10.489 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:25:10.489 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-25 16:40:36.345756448 +0000 2024-03-25T18:25:10.489 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-25 16:32:16.719250259 +0000 2024-03-25T18:25:10.489 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-25 16:32:16.719250259 +0000 2024-03-25T18:25:10.489 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-25 16:32:16.719250259 +0000 2024-03-25T18:25:10.489 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:25:10.554 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-25T18:25:10.554 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-25T18:25:10.555 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000174572 s, 2.9 MB/s 2024-03-25T18:25:10.556 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:25:10.612 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:10.612 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:25:10.612 INFO:tasks.ceph:Generating config... 2024-03-25T18:25:10.613 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:25:10.614 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:25:10.614 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:25:10.614 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:25:10.614 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:25:10.614 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:25:10.614 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:25:10.614 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:25:10.614 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:25:10.614 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:25:10.614 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:25:10.614 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:25:10.614 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:25:10.614 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:25:10.614 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:25:10.615 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:25:10.615 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:25:10.615 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:25:10.615 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:25:10.615 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:25:10.615 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:25:10.615 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:25:10.615 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:25:10.615 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:25:10.615 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:25:10.615 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:25:10.615 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:25:10.615 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:25:10.615 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:25:10.615 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:25:10.615 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:25:10.616 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:25:10.617 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:25:10.617 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:25:10.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:25:10.678 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:25:10.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:25:10.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:25:10.752 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.102:6789]'), ('mon.c', '[v1:172.21.15.102:6790]'), ('mon.b', '[v1:172.21.15.146:6789]')] 2024-03-25T18:25:10.752 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.102:6789],[v1:172.21.15.102:6790],[v1:172.21.15.146:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:25:10.753 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:10.753 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:25:10.807 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.102:6789]' --addv c '[v1:172.21.15.102:6790]' --addv b '[v1:172.21.15.146:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:10.891 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: generated fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = quincy 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 0 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:min_mon_release 17 (quincy) 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:election_strategy: 3 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:25:10.892 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:25:10.893 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:25:10.908 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 56a4cafe-eb50-4500-9868-7090e2ea20fc... 2024-03-25T18:25:10.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:25:10.951 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: chdir = "" 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: auth supported = cephx 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: filestore xattr use omap = true 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: mon clock drift allowed = 1.000 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: osd crush chooseleaf type = 0 2024-03-25T18:25:10.998 INFO:teuthology.orchestra.run.smithi102.stdout: auth debug = true 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on old message = true 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on bug = true 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: mon pg warn max object skew = 0 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default size = 2 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow primary affinity = true 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow pg remap = true 2024-03-25T18:25:10.999 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on no sortbitwise = false 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on too few osds = false 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon_allow_pool_size_one = true 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: osd default data pool replay window = 5 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon allow pool delete = true 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: mon cluster log file level = debug 2024-03-25T18:25:11.000 INFO:teuthology.orchestra.run.smithi102.stdout: debug asserts on shutdown = true 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: mon health detail to clog = false 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: mon host = "[v1:172.21.15.102:6789],[v1:172.21.15.102:6790],[v1:172.21.15.146:6789]" 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: mon client directed command retry = 5 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: mon election default strategy = 3 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: ms bind msgr2 = False 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: ms inject socket failures = 5000 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: ms type = async 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: osd heartbeat grace = 80 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout:[osd] 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: osd journal size = 100 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub load threshold = 5.0 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub max interval = 600 2024-03-25T18:25:11.001 INFO:teuthology.orchestra.run.smithi102.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd recover clone overlap = true 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd recovery max chunk = 1048576 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug shutdown = true 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug op order = true 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify stray on activate = true 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug trim objects = true 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd open classes on start = true 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug pg log writeout = true 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:25:11.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: osd map max advance = 10 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: journal zero on create = true 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: filestore apply finisher threads = 3 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: bdev debug aio = true 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug misdirected ops = true 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: bdev async discard = True 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: bdev enable discard = True 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore allocator = stupid 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore block size = 96636764160 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore fsck on mount = True 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore zero block detection = True 2024-03-25T18:25:11.003 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluefs = 1 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluestore = 1 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: debug monc = 20 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: debug osd = 20 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: debug rocksdb = 10 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd full ratio = 0.9 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: osd class default list = * 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: osd class load list = * 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: osd client watch timeout = 120 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify cached snaps = True 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify missing on start = True 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:25:11.004 INFO:teuthology.orchestra.run.smithi102.stdout: osd fast shutdown = False 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: osd max markdown count = 1000 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: osd mclock override recovery settings = True 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: osd objectstore = bluestore 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: osd op queue = debug_random 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: osd op queue cut off = debug_random 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: osd shutdown pgref assert = True 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr] 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgr = 20 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: mgr/telemetry/nag = false 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.005 INFO:teuthology.orchestra.run.smithi102.stdout:[mon] 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: debug paxos = 20 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: mon data avail warn = 5 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: mon mgr mkfs grace = 240 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd reporter subtree level = osd 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd prime pg temp = true 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:25:11.006 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: # 1m isn't quite enough 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon min osdmap epochs = 50 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd crush smoke test = False 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon osdmap full prune interval = 2 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon osdmap full prune min = 15 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon scrub interval = 300 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on pool no app = False 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: paxos service trim min = 10 2024-03-25T18:25:11.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout:[client] 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: rgw cache enabled = true 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable ops log = true 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable usage log = true 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: debug monc = 20 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: debug objecter = 20 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout: debug rados = 20 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.a] 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.c] 2024-03-25T18:25:11.008 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.b] 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-25T18:25:11.009 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:25:11.010 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "[v1:172.21.15.102:6789],[v1:172.21.15.102:6790],[v1:172.21.15.146:6789]" 2024-03-25T18:25:11.011 INFO:teuthology.orchestra.run.smithi146.stdout: mon client directed command retry = 5 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: mon election default strategy = 3 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: ms bind msgr2 = False 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject socket failures = 5000 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: ms type = async 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: osd heartbeat grace = 80 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-25T18:25:11.012 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.013 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = stupid 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore zero block detection = True 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: debug monc = 20 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-25T18:25:11.014 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 10 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd class default list = * 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd class load list = * 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd client watch timeout = 120 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify cached snaps = True 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify missing on start = True 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd fast shutdown = False 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd max markdown count = 1000 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock override recovery settings = True 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-25T18:25:11.015 INFO:teuthology.orchestra.run.smithi146.stdout: osd op queue = debug_random 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: osd op queue cut off = debug_random 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: osd shutdown pgref assert = True 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-25T18:25:11.016 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:25:11.017 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon min osdmap epochs = 50 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd crush smoke test = False 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune interval = 2 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune min = 15 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon scrub interval = 300 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on pool no app = False 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: paxos service trim min = 10 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-25T18:25:11.018 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: debug monc = 20 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: debug objecter = 20 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout: debug rados = 20 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-25T18:25:11.019 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-25T18:25:11.020 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:25:11.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:25:11.075 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:25:11.075 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:11.075 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:25:11.089 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:11.089 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:25:11.144 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:25:11.144 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:11.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:25:11.144 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:25:11.223 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:11.223 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:11.278 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:25:11.278 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:11.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:25:11.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:25:11.319 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:11.320 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:11.376 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:25:11.376 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:25:11.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:25:11.430 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:25:11.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:25:11.501 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:25:11.504 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:25:11.504 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:25:11.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:25:11.546 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:25:11.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:25:11.603 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:25:11.618 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:25:11.618 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:25:11.618 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:25:11.646 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:11.646 INFO:tasks.ceph:role: osd.0 2024-03-25T18:25:11.646 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:25:11.647 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:11.721 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:12.242 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-25T18:25:12.244 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-25T18:25:12.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:25:12.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:25:12.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:25:12.467 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:12.468 INFO:tasks.ceph:role: osd.1 2024-03-25T18:25:12.468 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:25:12.468 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:25:12.542 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:12.542 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:12.542 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:12.542 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:12.542 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:12.543 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:12.543 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:12.543 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:12.995 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-25T18:25:12.997 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-25T18:25:12.997 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:25:13.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:25:13.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:25:13.191 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:13.191 INFO:tasks.ceph:role: osd.2 2024-03-25T18:25:13.191 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:25:13.191 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:25:13.265 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:13.265 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:13.265 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:13.265 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:13.265 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:13.266 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:13.266 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:13.266 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:13.266 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:13.266 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:13.751 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-25T18:25:13.753 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-25T18:25:13.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:25:13.876 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:25:13.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:25:13.974 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:13.974 INFO:tasks.ceph:role: osd.3 2024-03-25T18:25:13.974 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:25:13.974 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:14.049 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:14.557 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-25T18:25:14.559 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-25T18:25:14.559 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:25:14.683 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:25:14.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:14.818+0000 7f2782a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:25:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:14.818+0000 7f2782a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:25:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:14.818+0000 7f2782a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:25:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:14.819+0000 7f2782a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:25:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:14.819+0000 7f2782a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:25:14.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:14.819+0000 7f2782a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:25:17.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:25:17.695 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:17.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:17.784+0000 7f92422a0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:25:17.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:17.785+0000 7f92422a0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:25:17.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:17.785+0000 7f92422a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:25:17.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:17.785+0000 7f92422a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:25:17.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:17.785+0000 7f92422a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:25:17.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:17.785+0000 7f92422a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:25:20.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:25:20.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:20.749+0000 7f32fcc68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:25:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:20.749+0000 7f32fcc68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:25:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:20.749+0000 7f32fcc68640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:25:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:20.749+0000 7f32fcc68640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:25:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:20.749+0000 7f32fcc68640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:25:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:20.749+0000 7f32fcc68640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:25:23.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:25:23.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:23.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:23.722+0000 7fd564a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:25:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:23.722+0000 7fd564a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:25:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:23.722+0000 7fd564a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:25:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:23.722+0000 7fd564a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:25:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:23.722+0000 7fd564a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:25:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:23.723+0000 7fd564a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:25:26.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:25:26.595 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:25:26.627 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:26.628 INFO:tasks.ceph:role: osd.4 2024-03-25T18:25:26.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:25:26.628 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:26.706 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:26.707 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:27.210 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-25T18:25:27.211 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-25T18:25:27.212 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:25:27.354 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:25:27.394 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:25:27.466 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:27.466 INFO:tasks.ceph:role: osd.5 2024-03-25T18:25:27.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:25:27.466 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:25:27.541 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:27.542 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:27.542 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:27.542 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:27.543 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:27.543 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:27.543 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:27.986 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-25T18:25:27.988 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-25T18:25:27.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:25:28.078 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:25:28.113 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:25:28.181 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:28.182 INFO:tasks.ceph:role: osd.6 2024-03-25T18:25:28.182 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:25:28.182 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:25:28.257 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:28.257 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:28.258 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:28.258 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:28.258 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:28.258 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:28.258 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:28.258 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:28.258 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:28.258 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:28.747 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-25T18:25:28.749 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-25T18:25:28.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:25:28.888 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:25:28.924 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:25:28.993 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:25:28.993 INFO:tasks.ceph:role: osd.7 2024-03-25T18:25:28.993 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:25:28.993 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:25:29.069 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:25:29.587 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-25T18:25:29.589 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-25T18:25:29.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:25:29.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:25:29.770 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:29.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:29.867+0000 7ff5f7411640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:25:29.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:29.867+0000 7ff5f7411640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:25:29.869 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:29.867+0000 7ff5f7411640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:25:29.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:29.868+0000 7ff5f7411640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:25:29.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:29.868+0000 7ff5f7411640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:25:29.870 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:29.868+0000 7ff5f7411640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:25:32.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:25:32.741 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:32.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:32.829+0000 7f9663e0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:25:32.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:32.829+0000 7f9663e0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:25:32.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:32.829+0000 7f9663e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:25:32.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:32.829+0000 7f9663e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:25:32.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:32.829+0000 7f9663e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:25:32.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:32.830+0000 7f9663e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:25:35.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:25:35.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:35.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:35.795+0000 7fb22d4b1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:25:35.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:35.795+0000 7fb22d4b1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:25:35.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:35.795+0000 7fb22d4b1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:25:35.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:35.795+0000 7fb22d4b1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:25:35.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:35.795+0000 7fb22d4b1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:25:35.797 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:35.796+0000 7fb22d4b1640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:25:38.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:25:38.683 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:38.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:38.772+0000 7fd80ac48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:25:38.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:38.772+0000 7fd80ac48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:25:38.774 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:38.772+0000 7fd80ac48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:25:38.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:38.773+0000 7fd80ac48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:25:38.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:38.773+0000 7fd80ac48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:25:38.775 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:25:38.773+0000 7fd80ac48640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:25:41.616 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:25:41.647 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:25:41.648 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:41.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:25:41.679 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:41.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:25:41.749 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:41.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:25:41.817 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:41.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:25:41.885 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:41.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:25:41.952 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:41.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:25:41.984 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:41.985 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:25:42.056 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:42.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:25:42.126 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:42.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:25:42.195 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:42.195 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:25:42.263 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:25:42.264 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:25:42.279 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:25:42.279 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:25:42.319 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:25:42.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:25:42.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:25:42.362 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.y] 2024-03-25T18:25:42.362 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCHwQFmV7ixGRAAyd1+GsrlQLpf3XSVLGUHxQ== 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.0] 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCKwQFm7mPlMBAAfWNzuhF3KDt2HwTllsHFzw== 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.1] 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCNwQFmfc7eLhAAfgUNbCFQjA0iagLdJu9K1Q== 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.2] 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCQwQFmWQ+/LBAApTZhC0oJUslVNhf4IhEpXA== 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.3] 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCTwQFmDnAbKxAAasG9oAJAA+siGAMfrYFzlQ== 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.x] 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCHwQFmk2vqHRAAU5BOeuxZShXKmBPrO5+vLw== 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.4] 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCZwQFm8XnNMxAA2cb0lVTiHPm6swXPYE/DQg== 2024-03-25T18:25:42.363 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.5] 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCcwQFm8VmCMRAA3hTAHrFtDwX1lGLPaiiFIQ== 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.6] 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCfwQFm88d/LxAAo2IFd/yzjNvy8XF9Efn5Pw== 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.7] 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCiwQFmZ1gkLhAAzkyUgIdAIGJDAa279dvjmQ== 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCHwQFmJKeXIBAAUC5FNhR5MdclqPGMoE00WA== 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout:[client.1] 2024-03-25T18:25:42.364 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCHwQFmGT35IxAAd9fgiqbkNaEK1VcbVc1YtQ== 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.y] 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCHwQFmV7ixGRAAyd1+GsrlQLpf3XSVLGUHxQ== 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCKwQFm7mPlMBAAfWNzuhF3KDt2HwTllsHFzw== 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCNwQFmfc7eLhAAfgUNbCFQjA0iagLdJu9K1Q== 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCQwQFmWQ+/LBAApTZhC0oJUslVNhf4IhEpXA== 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCTwQFmDnAbKxAAasG9oAJAA+siGAMfrYFzlQ== 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-03-25T18:25:42.386 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCHwQFmk2vqHRAAU5BOeuxZShXKmBPrO5+vLw== 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCZwQFm8XnNMxAA2cb0lVTiHPm6swXPYE/DQg== 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCcwQFm8VmCMRAA3hTAHrFtDwX1lGLPaiiFIQ== 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.6] 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCfwQFm88d/LxAAo2IFd/yzjNvy8XF9Efn5Pw== 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.7] 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCiwQFmZ1gkLhAAzkyUgIdAIGJDAa279dvjmQ== 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCHwQFmJKeXIBAAUC5FNhR5MdclqPGMoE00WA== 2024-03-25T18:25:42.387 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-03-25T18:25:42.388 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCHwQFmGT35IxAAd9fgiqbkNaEK1VcbVc1YtQ== 2024-03-25T18:25:42.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:25:42.408 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:25:42.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.539 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.647 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.698 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:25:42.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:25:42.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.755 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.810 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:25:42.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:25:42.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:25:43.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:25:43.029 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:25:43.083 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:25:43.083 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:25:43.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:25:43.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:25:43.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:25:43.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:25:43.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:25:43.420 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:25:43.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:25:43.547 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:25:43.574 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:43.577 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:25:43.628 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:25:43.629 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:25:43.629 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:25:43.629 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:25:43.631 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:25:43.631 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:25:43.631 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:25:43.632 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:25:43.633 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:25:43.633 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:25:43.634 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:25:43.634 DEBUG:teuthology.orchestra.run.smithi146:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:25:43.670 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:25:43.671 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:25:43.671 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:25:43.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:25:43.673 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:25:43.674 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:25:43.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:25:43.676 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:25:43.676 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:25:43.676 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.783+0000 7fac89764640 10 monclient: get_monmap_and_config 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.783+0000 7fac89764640 10 monclient: build_initial_monmap 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.783+0000 7fac89764640 10 monclient: monmap: 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:25:43.783961+0000 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:25:43.783961+0000 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:25:43.784 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:25:43.785 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:25:43.785 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:25:43.785 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:25:43.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.783+0000 7fac89764640 1 Processor -- start 2024-03-25T18:25:43.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.783+0000 7fac89764640 1 -- start start 2024-03-25T18:25:43.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.783+0000 7fac89764640 10 monclient: init 2024-03-25T18:25:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:43.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 10 monclient(hunting): picked mon.noname-a con 0x7fac8406c2d0 addr v1:172.21.15.102:6789/0 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 10 monclient(hunting): picked mon.noname-c con 0x7fac8406c6b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 10 monclient(hunting): picked mon.noname-b con 0x7fac84142d90 addr v1:172.21.15.102:6790/0 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840b0840 con 0x7fac8406c2d0 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac84143200 con 0x7fac8406c6b0 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8406d810 con 0x7fac84142d90 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac837fe640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac8414dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac8406c2d0 legacy=0x7fac840afde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac89764640 10 monclient(hunting): authenticate will time out at 7375.072508s 2024-03-25T18:25:43.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.784+0000 7fac827fc640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac84142680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.985+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac8406c2d0 legacy=0x7fac840afde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.985+0000 7fac837fe640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac8414dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:43.985+0000 7fac827fc640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac84142680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:44.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:44.386+0000 7fac837fe640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac8414dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:44.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:44.386+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac8406c2d0 legacy=0x7fac840afde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:44.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:44.386+0000 7fac827fc640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac84142680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:45.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:45.187+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac8406c2d0 legacy=0x7fac840afde0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:45.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:45.187+0000 7fac837fe640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac8414dbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:45.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:45.187+0000 7fac827fc640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac84142680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:46.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 10 monclient(hunting): tick 2024-03-25T18:25:46.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:46.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac8414dbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac84142680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac8406c2d0 legacy=0x7fac840afde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fac70003680 addr v1:172.21.15.102:6789/0 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fac84142d90 addr v1:172.21.15.102:6790/0 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fac8406c6b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840ae260 con 0x7fac70003680 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840b0840 con 0x7fac8406c6b0 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8406d810 con 0x7fac84142d90 2024-03-25T18:25:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac80ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:46.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac70005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:46.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac827fc640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac70009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:46.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.784+0000 7fac837fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:46.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.985+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac70005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:46.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.985+0000 7fac827fc640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac70009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:46.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:46.985+0000 7fac837fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:47.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:47.386+0000 7fac827fc640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac70009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:47.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:47.386+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac70005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:47.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:47.386+0000 7fac837fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:48.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:48.187+0000 7fac827fc640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac70009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:48.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:48.187+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac70005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:48.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:48.187+0000 7fac837fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:49.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 10 monclient(hunting): tick 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac70009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac70005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fac84142d90 addr v1:172.21.15.102:6790/0 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fac70003680 addr v1:172.21.15.102:6789/0 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fac8406c6b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:25:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac70011430 con 0x7fac70003680 2024-03-25T18:25:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840ae260 con 0x7fac8406c6b0 2024-03-25T18:25:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840b0840 con 0x7fac84142d90 2024-03-25T18:25:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac80ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac827fc640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac7000a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac700103a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.784+0000 7fac837fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:49.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.985+0000 7fac827fc640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac7000a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:49.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.985+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac700103a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:49.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:49.986+0000 7fac837fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:50.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:50.386+0000 7fac827fc640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac7000a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:50.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:50.386+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac700103a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:50.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:50.387+0000 7fac837fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:50.658 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:00:06.871 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:50.944 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:00:07.180 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:50.947 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:00:07.183 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:50.965 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:00:07.179 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:51.187+0000 7fac827fc640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac7000a310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-25T18:25:51.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:51.187+0000 7fac82ffd640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac700103a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6790/0 2024-03-25T18:25:51.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:51.188+0000 7fac837fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-03-25T18:25:51.264 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:00:07.500 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:51.267 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:00:07.503 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 10 monclient(hunting): tick 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac700103a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac7000a310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fac70003680 addr v1:172.21.15.102:6789/0 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fac84142d90 addr v1:172.21.15.102:6790/0 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fac8406c6b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8406d810 con 0x7fac70003680 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac70011430 con 0x7fac8406c6b0 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840ae260 con 0x7fac84142d90 2024-03-25T18:25:52.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.784+0000 7fac80ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:52.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.838+0000 7fac837fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 0x7fac7000cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:41566/0 (socket says 172.21.15.102:41566) 2024-03-25T18:25:52.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:52.838+0000 7fac837fe640 1 -- 172.21.15.102:0/2241485059 learned_addr learned my addr 172.21.15.102:0/2241485059 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:25:55.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 10 monclient(hunting): tick 2024-03-25T18:25:55.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:55.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:55.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:55.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac70010370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.146:6789/0 conn(0x7fac8406c6b0 legacy=0x7fac7000cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac70010b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fac70005b00 addr v1:172.21.15.146:6789/0 2024-03-25T18:25:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fac84142d90 addr v1:172.21.15.102:6790/0 2024-03-25T18:25:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fac70003680 addr v1:172.21.15.102:6789/0 2024-03-25T18:25:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac840b0840 con 0x7fac70003680 2024-03-25T18:25:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac70005fe0 con 0x7fac70005b00 2024-03-25T18:25:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7000cbe0 con 0x7fac84142d90 2024-03-25T18:25:55.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:55.784+0000 7fac80ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 10 monclient(hunting): tick 2024-03-25T18:25:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:25:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:25:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:25:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac70010370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.146:6789/0 conn(0x7fac70005b00 legacy=0x7fac7000d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.102:6789/0 conn(0x7fac70003680 legacy=0x7fac70010b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fac7000d9d0 addr v1:172.21.15.146:6789/0 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fac7000de10 addr v1:172.21.15.102:6789/0 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fac84142d90 addr v1:172.21.15.102:6790/0 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7000ce10 con 0x7fac7000de10 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac70010b50 con 0x7fac7000d9d0 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac70010ce0 con 0x7fac84142d90 2024-03-25T18:25:58.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:25:58.784+0000 7fac80ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:01.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.187+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331095483 0 0) 0x7fac70010ce0 con 0x7fac84142d90 2024-03-25T18:26:01.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.187+0000 7fac81ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.187+0000 7fac81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.187+0000 7fac81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.187+0000 7fac81ffb640 10 monclient(hunting): my global_id is 4103 2024-03-25T18:26:01.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.187+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac6c003520 con 0x7fac84142d90 2024-03-25T18:26:01.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.217+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264440451 0 0) 0x7fac70010b50 con 0x7fac7000d9d0 2024-03-25T18:26:01.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.217+0000 7fac81ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.217+0000 7fac81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.217+0000 7fac81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.217+0000 7fac81ffb640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:26:01.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.217+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac70010ce0 con 0x7fac7000d9d0 2024-03-25T18:26:01.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.222+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539365370 0 0) 0x7fac7000ce10 con 0x7fac7000de10 2024-03-25T18:26:01.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.222+0000 7fac81ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.222+0000 7fac81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.222+0000 7fac81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.222+0000 7fac81ffb640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:26:01.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.222+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac70010b50 con 0x7fac7000de10 2024-03-25T18:26:01.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1220502228 0 0) 0x7fac6c003520 con 0x7fac84142d90 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.146:6789/0 conn(0x7fac7000d9d0 legacy=0x7fac700092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.102:6789/0 conn(0x7fac7000de10 legacy=0x7fac700156c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 1 monclient: found mon.noname-b 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac70010f10 con 0x7fac84142d90 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fac7c004400 con 0x7fac84142d90 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:01.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac81ffb640 10 monclient: dump: 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac89764640 5 monclient: authenticate success, global_id 4103 2024-03-25T18:26:01.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.254+0000 7fac89764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.293+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac7c0045d0 con 0x7fac84142d90 2024-03-25T18:26:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.293+0000 7fac81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.293+0000 7fac81ffb640 1 -- 172.21.15.102:0/2241485059 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fac7c0049e0 con 0x7fac84142d90 2024-03-25T18:26:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.293+0000 7fac81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.293+0000 7fac81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.293+0000 7fac81ffb640 10 monclient: dump: 2024-03-25T18:26:01.295 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.293+0000 7fac89764640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:01.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.293+0000 7fac89764640 4 set_mon_vals no callback set 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.294+0000 7fac89764640 10 monclient: shutdown 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.294+0000 7fac89764640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.294+0000 7fac89764640 1 -- 172.21.15.102:0/2241485059 >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac7000d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.294+0000 7fac89764640 1 -- 172.21.15.102:0/2241485059 shutdown_connections 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.294+0000 7fac89764640 1 -- 172.21.15.102:0/2241485059 >> 172.21.15.102:0/2241485059 conn(0x7fac840b5e40 msgr2=0x7fac840b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.294+0000 7fac89764640 1 -- 172.21.15.102:0/2241485059 shutdown_connections 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.294+0000 7fac89764640 1 -- 172.21.15.102:0/2241485059 wait complete. 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 10 monclient: build_initial_monmap 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 10 monclient: monmap: 2024-03-25T18:26:01.299 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 1 librados: starting msgr at 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 1 librados: starting objecter 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 1 Processor -- start 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 1 -- start start 2024-03-25T18:26:01.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 1 librados: setting wanted keys 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 1 librados: calling monclient init 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.295+0000 7fac89764640 10 monclient: init 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 10 monclient(hunting): picked mon.noname-a con 0x7fac7000d9d0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 10 monclient(hunting): picked mon.noname-b con 0x7fac7000de10 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 10 monclient(hunting): picked mon.noname-c con 0x7fac84142d90 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8413dd00 con 0x7fac7000d9d0 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8413df50 con 0x7fac7000de10 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac841442d0 con 0x7fac84142d90 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.296+0000 7fac89764640 10 monclient(hunting): authenticate will time out at 7392.585384s 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.297+0000 7fac827fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fac7000de10 0x7fac8413d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:41630/0 (socket says 172.21.15.102:41630) 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.297+0000 7fac827fc640 1 -- 172.21.15.102:0/1336966528 learned_addr learned my addr 172.21.15.102:0/1336966528 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:01.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.297+0000 7fac6b7fe640 10 client.?.objecter ms_handle_connect 0x7fac7000de10 2024-03-25T18:26:01.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.297+0000 7fac6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.312+0000 7fac6b7fe640 10 client.?.objecter ms_handle_connect 0x7fac7000d9d0 2024-03-25T18:26:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.312+0000 7fac6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:01.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.339+0000 7fac6b7fe640 10 client.?.objecter ms_handle_connect 0x7fac84142d90 2024-03-25T18:26:01.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.339+0000 7fac6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.343+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815170327 0 0) 0x7fac841442d0 con 0x7fac84142d90 2024-03-25T18:26:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.344+0000 7fac6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.344+0000 7fac6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.344+0000 7fac6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.344+0000 7fac6b7fe640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:26:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.344+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac700110a0 con 0x7fac84142d90 2024-03-25T18:26:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.357+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432053970 0 0) 0x7fac8413df50 con 0x7fac7000de10 2024-03-25T18:26:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.357+0000 7fac6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.357+0000 7fac6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.357+0000 7fac6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.357+0000 7fac6b7fe640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:26:01.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.357+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac841442d0 con 0x7fac7000de10 2024-03-25T18:26:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.363+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816370563 0 0) 0x7fac8413dd00 con 0x7fac7000d9d0 2024-03-25T18:26:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.363+0000 7fac6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.363+0000 7fac6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.363+0000 7fac6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.363+0000 7fac6b7fe640 10 monclient(hunting): my global_id is 4110 2024-03-25T18:26:01.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.363+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac8413df50 con 0x7fac7000d9d0 2024-03-25T18:26:01.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.366+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3713966309 0 0) 0x7fac700110a0 con 0x7fac84142d90 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.366+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac8413dd00 con 0x7fac84142d90 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.366+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fac7c004740 con 0x7fac84142d90 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.366+0000 7fac6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.366+0000 7fac6b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.366+0000 7fac6b7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:01.368 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:01.369 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:01.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.366+0000 7fac6b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:01.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2915014353 0 0) 0x7fac8413dd00 con 0x7fac84142d90 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 >> v1:172.21.15.146:6789/0 conn(0x7fac7000de10 legacy=0x7fac8413d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 >> v1:172.21.15.102:6789/0 conn(0x7fac7000d9d0 legacy=0x7fac8413e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 1 monclient: found mon.c 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac840715c0 con 0x7fac84142d90 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 5 monclient: authenticate success, global_id 4109 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 10 monclient: _renew_subs 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac84071810 con 0x7fac84142d90 2024-03-25T18:26:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 10 client.4109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 10 monclient: _renew_subs 2024-03-25T18:26:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8413bee0 con 0x7fac84142d90 2024-03-25T18:26:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.432+0000 7fac89764640 1 librados: init done 2024-03-25T18:26:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.434+0000 7fac89764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.434+0000 7fac89764640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.434+0000 7fac89764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.434+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac84144460 con 0x7fac84142d90 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac7c003ff0 con 0x7fac84142d90 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fac7c004df0 con 0x7fac84142d90 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 10 monclient: dump: 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:01.648 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fac7c0053a0 con 0x7fac84142d90 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3842137267 0 0) 0x7fac7c0069a0 con 0x7fac84142d90 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 10 client.4109.objecter ms_dispatch 0x7fac84000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 3 client.4109.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac817fa640 4 set_mon_vals no callback set 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 3 client.4109.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:26:01.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 20 client.4109.objecter dump_active .. 0 homeless 2024-03-25T18:26:01.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fac7c018de0 con 0x7fac84142d90 2024-03-25T18:26:01.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:01.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.646+0000 7fac6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:01.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.755+0000 7fac89764640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:26:01.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.755+0000 7fac89764640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:26:01.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.755+0000 7fac89764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:01.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.755+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fac8404f730 con 0x7fac84142d90 2024-03-25T18:26:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.929+0000 7fac6b7fe640 1 -- 172.21.15.102:0/1336966528 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fac7c005730 con 0x7fac84142d90 2024-03-25T18:26:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.929+0000 7fac6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:26:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.929+0000 7fac6b7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:26:01.931 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-03-25T18:26:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 10 librados: watch_flush enter 2024-03-25T18:26:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 10 librados: watch_flush exit 2024-03-25T18:26:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 20 client.4109.objecter shutdown clearing up homeless session... 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 10 client.4109.objecter successfully canceled tick 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 10 monclient: shutdown 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 >> v1:172.21.15.102:6790/0 conn(0x7fac84142d90 legacy=0x7fac841e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 shutdown_connections 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 >> 172.21.15.102:0/1336966528 conn(0x7fac840b5e40 msgr2=0x7fac840b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 shutdown_connections 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 1 -- 172.21.15.102:0/1336966528 wait complete. 2024-03-25T18:26:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:01.930+0000 7fac89764640 1 librados: shutdown 2024-03-25T18:26:01.939 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:01.938+0000 7f51dce2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:26:01.942 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:26:01.942 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:26:01.942 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:26:01.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:26:01.969 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:26:01.970 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:26:01.993 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:01.992+0000 7f51dce2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:26:02.038 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:26:02.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:26:02.106 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-25T18:26:02.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:26:02.174 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:26:02.174 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:26:02.201 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:26:02.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:26:02.259 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:02.258+0000 7f9213610040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:26:02.268 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:26:02.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:26:02.316 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:02.315+0000 7f9213610040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:26:02.338 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-25T18:26:02.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:26:02.406 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new d665d3c6-ac37-40ea-b391-93ce0dd4500b 0 2024-03-25T18:26:02.537 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:02.537+0000 7f51dce2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:26:02.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.549+0000 7f501645c640 10 monclient: get_monmap_and_config 2024-03-25T18:26:02.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.549+0000 7f501645c640 10 monclient: build_initial_monmap 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.549+0000 7f501645c640 10 monclient: monmap: 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:26:02.549807+0000 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:26:02.549807+0000 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:02.550 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:02.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 1 Processor -- start 2024-03-25T18:26:02.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 1 -- start start 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 10 monclient: init 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 10 monclient(hunting): picked mon.noname-b con 0x7f50100a9920 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 10 monclient(hunting): picked mon.noname-a con 0x7f50100b45a0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 10 monclient(hunting): picked mon.noname-c con 0x7f5010146450 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501006b100 con 0x7f50100b45a0 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50100b2940 con 0x7f5010146450 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501013fec0 con 0x7f50100a9920 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.550+0000 7f501645c640 10 monclient(hunting): authenticate will time out at 7394.918098s 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.551+0000 7f50149f9640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5010146450 0x7f501014dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49394/0 (socket says 172.21.15.146:49394) 2024-03-25T18:26:02.552 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.551+0000 7f50149f9640 1 -- 172.21.15.146:0/2900542101 learned_addr learned my addr 172.21.15.146:0/2900542101 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:02.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069727469 0 0) 0x7f50100b2940 con 0x7f5010146450 2024-03-25T18:26:02.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f500effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f500effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f500effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f500effd640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff8003520 con 0x7f5010146450 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f50149f9640 1 -- 172.21.15.146:0/2900542101 >> v1:172.21.15.146:6789/0 conn(0x7f5010146450 legacy=0x7f501014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f50149f9640 1 -- 172.21.15.146:0/2900542101 >> v1:172.21.15.146:6789/0 conn(0x7f5010146450 legacy=0x7f501014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f50149f9640 1 --1- 172.21.15.146:0/2900542101 >> v1:172.21.15.146:6789/0 conn(0x7f5010146450 0x7f501014dc70 :-1 s=OPENED pgs=25 cs=1 l=1).handle_message read tag failed 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f50149f9640 1 --1- 172.21.15.146:0/2900542101 >> v1:172.21.15.146:6789/0 conn(0x7f5010146450 0x7f501014dc70 :-1 s=OPENED pgs=25 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:26:02.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.554+0000 7f500effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.146:6789/0 2024-03-25T18:26:02.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.592+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294956380 0 0) 0x7f501006b100 con 0x7f50100b45a0 2024-03-25T18:26:02.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.592+0000 7f500effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.592+0000 7f500effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.592+0000 7f500effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.592+0000 7f500effd640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:26:02.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.592+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50100b2940 con 0x7f50100b45a0 2024-03-25T18:26:02.594 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:02.594+0000 7f51dce2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1279310411 0 0) 0x7f50100b2940 con 0x7f50100b45a0 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 >> v1:172.21.15.102:6790/0 conn(0x7f50100a9920 legacy=0x7f501013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 >> v1:172.21.15.146:6789/0 conn(0x7f5010146450 legacy=0x7f501014dc70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 1 monclient: found mon.noname-a 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5010140050 con 0x7f50100b45a0 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:02.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f501645c640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f501645c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f5004001490 con 0x7f50100b45a0 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f500effd640 10 monclient: dump: 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:02.597 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:02.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.595+0000 7f501645c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.597+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5004001720 con 0x7f50100b45a0 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.597+0000 7f500effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.597+0000 7f501645c640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.597+0000 7f501645c640 4 set_mon_vals no callback set 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.597+0000 7f500effd640 1 -- 172.21.15.146:0/2900542101 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f5004001ab0 con 0x7f50100b45a0 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.597+0000 7f500effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.597+0000 7f500effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.597+0000 7f500effd640 10 monclient: dump: 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.599 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.598+0000 7f501645c640 10 monclient: shutdown 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.598+0000 7f501645c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.598+0000 7f501645c640 1 -- 172.21.15.146:0/2900542101 >> v1:172.21.15.102:6789/0 conn(0x7f50100b45a0 legacy=0x7f5010145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.598+0000 7f501645c640 1 -- 172.21.15.146:0/2900542101 shutdown_connections 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.598+0000 7f501645c640 1 -- 172.21.15.146:0/2900542101 >> 172.21.15.146:0/2900542101 conn(0x7f50100b7f40 msgr2=0x7f50100b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.598+0000 7f501645c640 1 -- 172.21.15.146:0/2900542101 shutdown_connections 2024-03-25T18:26:02.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.598+0000 7f501645c640 1 -- 172.21.15.146:0/2900542101 wait complete. 2024-03-25T18:26:02.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.599+0000 7f501645c640 10 monclient: build_initial_monmap 2024-03-25T18:26:02.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.599+0000 7f501645c640 10 monclient: monmap: 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.599+0000 7f501645c640 1 librados: starting msgr at 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.599+0000 7f501645c640 1 librados: starting objecter 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.599+0000 7f501645c640 1 Processor -- start 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 1 -- start start 2024-03-25T18:26:02.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 1 librados: setting wanted keys 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 1 librados: calling monclient init 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 10 monclient: init 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 10 monclient(hunting): picked mon.noname-c con 0x7f50100a9920 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 10 monclient(hunting): picked mon.noname-a con 0x7f50100b45a0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 10 monclient(hunting): picked mon.noname-b con 0x7f5010146450 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.600+0000 7f501645c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50101d7390 con 0x7f50100b45a0 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f501645c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50101d7f30 con 0x7f5010146450 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f501645c640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50101d80c0 con 0x7f50100a9920 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f501645c640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f501645c640 10 monclient(hunting): authenticate will time out at 7394.968359s 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f50149f9640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5010146450 0x7f50101e1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49398/0 (socket says 172.21.15.146:49398) 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f50149f9640 1 -- 172.21.15.146:0/976471701 learned_addr learned my addr 172.21.15.146:0/976471701 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:02.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f500cff9640 10 client.?.objecter ms_handle_connect 0x7f5010146450 2024-03-25T18:26:02.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f500cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f500cff9640 10 client.?.objecter ms_handle_connect 0x7f50100b45a0 2024-03-25T18:26:02.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.601+0000 7f500cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.603+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464124765 0 0) 0x7f50101d7390 con 0x7f50100b45a0 2024-03-25T18:26:02.605 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.603+0000 7f500cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.603+0000 7f500cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.603+0000 7f500cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.603+0000 7f500cff9640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:26:02.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.603+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ffc004eb0 con 0x7f50100b45a0 2024-03-25T18:26:02.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.604+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943302296 0 0) 0x7f50101d7f30 con 0x7f5010146450 2024-03-25T18:26:02.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.604+0000 7f500cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.604+0000 7f500cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.604+0000 7f500cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.604+0000 7f500cff9640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:26:02.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.604+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50101d7390 con 0x7f5010146450 2024-03-25T18:26:02.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.608+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3117785853 0 0) 0x7f4ffc004eb0 con 0x7f50100b45a0 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.608+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50101d7f30 con 0x7f50100b45a0 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.608+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f5004001490 con 0x7f50100b45a0 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.608+0000 7f500cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.608+0000 7f500cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.608+0000 7f500cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:02.609 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:02.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.608+0000 7f500cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:02.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226376077 0 0) 0x7f50101d7390 con 0x7f5010146450 2024-03-25T18:26:02.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ffc004eb0 con 0x7f5010146450 2024-03-25T18:26:02.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f5008001ac0 con 0x7f5010146450 2024-03-25T18:26:02.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:02.612 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1317375825 0 0) 0x7f50101d7f30 con 0x7f50100b45a0 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 >> v1:172.21.15.102:6790/0 conn(0x7f50100a9920 legacy=0x7f50101d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.611+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 >> v1:172.21.15.146:6789/0 conn(0x7f5010146450 legacy=0x7f50101e1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f500cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f500cff9640 1 monclient: found mon.a 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f500cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f500cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5010140f40 con 0x7f50100b45a0 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f500cff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f500cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:02.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f500cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 10 monclient: _renew_subs 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50101411f0 con 0x7f50100b45a0 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 10 monclient: _renew_subs 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5010142610 con 0x7f50100b45a0 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 1 librados: init done 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:02.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.612+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501013fe10 con 0x7f50100b45a0 2024-03-25T18:26:02.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.615+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50040038a0 con 0x7f50100b45a0 2024-03-25T18:26:02.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.615+0000 7f500cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:02.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.615+0000 7f500e7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:02.641 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:02.641+0000 7f51dce2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:26:02.696 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:02.696+0000 7f51dce2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:26:02.741 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:02.741+0000 7f51dce2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:26:02.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f5004003a10 con 0x7f50100b45a0 2024-03-25T18:26:02.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:02.778 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 10 monclient: dump: 2024-03-25T18:26:02.778 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5004003f00 con 0x7f50100b45a0 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2225478437 0 0) 0x7f5004005330 con 0x7f50100b45a0 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 10 client.4119.objecter ms_dispatch 0x7f5010000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:26:02.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:26:02.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.777+0000 7f500cff9640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:26:02.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.779+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f50040056b0 con 0x7f50100b45a0 2024-03-25T18:26:02.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.779+0000 7f500cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.779+0000 7f500cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:02.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.853+0000 7f501645c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d665d3c6-ac37-40ea-b391-93ce0dd4500b", "id": 0}] 2024-03-25T18:26:02.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.853+0000 7f501645c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d665d3c6-ac37-40ea-b391-93ce0dd4500b", "id": 0}] 2024-03-25T18:26:02.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.853+0000 7f501645c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:02.854 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:02.853+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d665d3c6-ac37-40ea-b391-93ce0dd4500b", "id": 0} v 0) v1 -- 0x7f50100b6370 con 0x7f50100b45a0 2024-03-25T18:26:02.864 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:02.864+0000 7f51dce2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:26:02.897 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:02.895+0000 7f9213610040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:26:02.938 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:02.938+0000 7f51dce2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:26:02.951 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:02.950+0000 7f9213610040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:26:02.985 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:02.985+0000 7f51dce2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:26:02.999 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:02.997+0000 7f9213610040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:26:03.039 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.039+0000 7f500cff9640 1 -- 172.21.15.146:0/976471701 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d665d3c6-ac37-40ea-b391-93ce0dd4500b", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3079905049 0 732707515) 0x7f5004005910 con 0x7f50100b45a0 2024-03-25T18:26:03.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.039+0000 7f500cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d665d3c6-ac37-40ea-b391-93ce0dd4500b", "id": 0}] 2024-03-25T18:26:03.040 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.039+0000 7f500cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:03.040 INFO:teuthology.orchestra.run.smithi146.stdout:0 2024-03-25T18:26:03.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 10 librados: watch_flush enter 2024-03-25T18:26:03.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 10 librados: watch_flush exit 2024-03-25T18:26:03.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:26:03.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:26:03.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 10 monclient: shutdown 2024-03-25T18:26:03.041 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:03.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 >> v1:172.21.15.102:6789/0 conn(0x7f50100b45a0 legacy=0x7f5010141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 shutdown_connections 2024-03-25T18:26:03.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 >> 172.21.15.146:0/976471701 conn(0x7f50100b7f40 msgr2=0x7f50100b7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:03.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 shutdown_connections 2024-03-25T18:26:03.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 1 -- 172.21.15.146:0/976471701 wait complete. 2024-03-25T18:26:03.042 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f501645c640 1 librados: shutdown 2024-03-25T18:26:03.042 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:03.040+0000 7f51dce2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:26:03.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new bb6f7c78-451d-4f40-b363-5de7e3f0fcb9 1 2024-03-25T18:26:03.055 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.053+0000 7f9213610040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:26:03.092 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:03.091+0000 7f51dce2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:26:03.097 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.095+0000 7f9213610040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:26:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.152+0000 7fde541af640 10 monclient: get_monmap_and_config 2024-03-25T18:26:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.152+0000 7fde541af640 10 monclient: build_initial_monmap 2024-03-25T18:26:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.152+0000 7fde541af640 10 monclient: monmap: 2024-03-25T18:26:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:26:03.152680+0000 2024-03-25T18:26:03.153 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:26:03.152680+0000 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.152+0000 7fde541af640 1 Processor -- start 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 1 -- start start 2024-03-25T18:26:03.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 10 monclient: init 2024-03-25T18:26:03.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:03.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:03.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 10 monclient(hunting): picked mon.noname-a con 0x7fde4c0a9fa0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 10 monclient(hunting): picked mon.noname-c con 0x7fde4c0ab520 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 10 monclient(hunting): picked mon.noname-b con 0x7fde4c0afc70 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c0ac500 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c0ab040 con 0x7fde4c0ab520 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c072a30 con 0x7fde4c0afc70 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.153+0000 7fde541af640 10 monclient(hunting): authenticate will time out at 7395.521045s 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.154+0000 7fde5174a640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fde4c0ab520 0x7fde4c13bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49404/0 (socket says 172.21.15.146:49404) 2024-03-25T18:26:03.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.154+0000 7fde5174a640 1 -- 172.21.15.146:0/3139223686 learned_addr learned my addr 172.21.15.146:0/3139223686 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:03.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692858616 0 0) 0x7fde4c0ab040 con 0x7fde4c0ab520 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde38003440 con 0x7fde4c0ab520 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815815228 0 0) 0x7fde4c0ac500 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c0ab040 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172390970 0 0) 0x7fde4c072a30 con 0x7fde4c0afc70 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:26:03.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.157+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c0ac500 con 0x7fde4c0afc70 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 646163910 0 0) 0x7fde4c0ac500 con 0x7fde4c0afc70 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 >> v1:172.21.15.146:6789/0 conn(0x7fde4c0ab520 legacy=0x7fde4c13bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 >> v1:172.21.15.102:6789/0 conn(0x7fde4c0a9fa0 legacy=0x7fde4c0af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 10 monclient: _finish_hunting 0 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 1 monclient: found mon.noname-b 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c0ac770 con 0x7fde4c0afc70 2024-03-25T18:26:03.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 10 monclient: _finish_auth 0 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 10 monclient: _check_auth_tickets 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fde44001930 con 0x7fde4c0afc70 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde50f49640 10 monclient: dump: 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.162 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.163 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde541af640 5 monclient: authenticate success, global_id 4118 2024-03-25T18:26:03.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.160+0000 7fde541af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.162+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde44001a90 con 0x7fde4c0afc70 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.162+0000 7fde50f49640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.162+0000 7fde50f49640 1 -- 172.21.15.146:0/3139223686 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fde44001ee0 con 0x7fde4c0afc70 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.162+0000 7fde50f49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.162+0000 7fde50f49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.162+0000 7fde50f49640 10 monclient: dump: 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.164 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.162+0000 7fde541af640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.162+0000 7fde541af640 4 set_mon_vals no callback set 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.163+0000 7fde541af640 10 monclient: shutdown 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.163+0000 7fde541af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.163+0000 7fde541af640 1 -- 172.21.15.146:0/3139223686 >> v1:172.21.15.102:6790/0 conn(0x7fde4c0afc70 legacy=0x7fde4c14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.163+0000 7fde541af640 1 -- 172.21.15.146:0/3139223686 shutdown_connections 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.163+0000 7fde541af640 1 -- 172.21.15.146:0/3139223686 >> 172.21.15.146:0/3139223686 conn(0x7fde4c06b560 msgr2=0x7fde4c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.163+0000 7fde541af640 1 -- 172.21.15.146:0/3139223686 shutdown_connections 2024-03-25T18:26:03.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.163+0000 7fde541af640 1 -- 172.21.15.146:0/3139223686 wait complete. 2024-03-25T18:26:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.164+0000 7fde541af640 10 monclient: build_initial_monmap 2024-03-25T18:26:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.164+0000 7fde541af640 10 monclient: monmap: 2024-03-25T18:26:03.167 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.164+0000 7fde541af640 1 librados: starting msgr at 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.164+0000 7fde541af640 1 librados: starting objecter 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.164+0000 7fde541af640 1 Processor -- start 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 1 -- start start 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 1 librados: setting wanted keys 2024-03-25T18:26:03.168 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 1 librados: calling monclient init 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 10 monclient: init 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 10 monclient(hunting): picked mon.noname-a con 0x7fde4c0a9fa0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 10 monclient(hunting): picked mon.noname-b con 0x7fde4c0afc70 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 10 monclient(hunting): picked mon.noname-c con 0x7fde4c138720 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c138b70 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.165+0000 7fde541af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c11c030 con 0x7fde4c0afc70 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.166+0000 7fde541af640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde4c11c2a0 con 0x7fde4c138720 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.166+0000 7fde541af640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.166+0000 7fde541af640 10 monclient(hunting): authenticate will time out at 7395.533383s 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.166+0000 7fde5274c640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fde4c138720 0x7fde4c070970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.146:44612/0 (socket says 172.21.15.146:44612) 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.166+0000 7fde5274c640 1 -- 172.21.15.146:0/265934307 learned_addr learned my addr 172.21.15.146:0/265934307 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.166+0000 7fde427fc640 10 client.?.objecter ms_handle_connect 0x7fde4c138720 2024-03-25T18:26:03.169 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.166+0000 7fde427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:03.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.167+0000 7fde427fc640 10 client.?.objecter ms_handle_connect 0x7fde4c0a9fa0 2024-03-25T18:26:03.170 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.167+0000 7fde427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.168+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017253929 0 0) 0x7fde4c11c2a0 con 0x7fde4c138720 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.168+0000 7fde427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.168+0000 7fde427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.168+0000 7fde427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.168+0000 7fde427fc640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.168+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde30004eb0 con 0x7fde4c138720 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.169+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32975518 0 0) 0x7fde4c138b70 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.169+0000 7fde427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.169+0000 7fde427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.169+0000 7fde427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.169+0000 7fde427fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.169+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c11c2a0 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.170+0000 7fde427fc640 10 client.?.objecter ms_handle_connect 0x7fde4c0afc70 2024-03-25T18:26:03.171 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.170+0000 7fde427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:03.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.172+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061430833 0 0) 0x7fde4c11c030 con 0x7fde4c0afc70 2024-03-25T18:26:03.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.172+0000 7fde427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.172+0000 7fde427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.172+0000 7fde427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.172+0000 7fde427fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:26:03.173 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.172+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c138b70 con 0x7fde4c0afc70 2024-03-25T18:26:03.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.174+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1522342156 0 0) 0x7fde30004eb0 con 0x7fde4c138720 2024-03-25T18:26:03.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.174+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde4c11c030 con 0x7fde4c138720 2024-03-25T18:26:03.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.174+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fde44001ea0 con 0x7fde4c138720 2024-03-25T18:26:03.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.174+0000 7fde427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.174+0000 7fde427fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:03.174 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.174+0000 7fde427fc640 10 monclient(hunting): dump: 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.175 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.174+0000 7fde427fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1727298033 0 0) 0x7fde4c11c2a0 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde30004eb0 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fde48001a50 con 0x7fde4c0a9fa0 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 10 monclient(hunting): dump: 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.177 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.178 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.178 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.179 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 165751699 0 0) 0x7fde4c11c030 con 0x7fde4c138720 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 >> v1:172.21.15.146:6789/0 conn(0x7fde4c0afc70 legacy=0x7fde4c138010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 >> v1:172.21.15.102:6789/0 conn(0x7fde4c0a9fa0 legacy=0x7fde4c1da9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 1 monclient: found mon.c 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c11c580 con 0x7fde4c138720 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde541af640 5 monclient: authenticate success, global_id 4121 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde541af640 10 monclient: _renew_subs 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde541af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.180 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4c1db520 con 0x7fde4c138720 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde541af640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde541af640 10 monclient: _renew_subs 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde541af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.177+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4c1db2f0 con 0x7fde4c138720 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.178+0000 7fde541af640 1 librados: init done 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.179+0000 7fde541af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.179+0000 7fde541af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.179+0000 7fde541af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.181 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.179+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4c04f730 con 0x7fde4c138720 2024-03-25T18:26:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde44003c20 con 0x7fde4c138720 2024-03-25T18:26:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fde44002100 con 0x7fde4c138720 2024-03-25T18:26:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:03.182 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 10 monclient: dump: 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fde44004400 con 0x7fde4c138720 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde43fff640 4 set_mon_vals no callback set 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2346036132 0 0) 0x7fde44006a00 con 0x7fde4c138720 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 10 client.4121.objecter ms_dispatch 0x7fde4c000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 3 client.4121.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:26:03.183 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 3 client.4121.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:26:03.184 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.181+0000 7fde427fc640 20 client.4121.objecter dump_active .. 0 homeless 2024-03-25T18:26:03.218 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.217+0000 7f9213610040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:26:03.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.278+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fde440046b0 con 0x7fde4c138720 2024-03-25T18:26:03.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.278+0000 7fde427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:03.279 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.278+0000 7fde427fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:03.293 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.292+0000 7f9213610040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:26:03.342 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.341+0000 7f9213610040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:26:03.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.370+0000 7fde541af640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bb6f7c78-451d-4f40-b363-5de7e3f0fcb9", "id": 1}] 2024-03-25T18:26:03.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.370+0000 7fde541af640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bb6f7c78-451d-4f40-b363-5de7e3f0fcb9", "id": 1}] 2024-03-25T18:26:03.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.370+0000 7fde541af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.370+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd new", "uuid": "bb6f7c78-451d-4f40-b363-5de7e3f0fcb9", "id": 1} v 0) v1 -- 0x7fde4c06a070 con 0x7fde4c138720 2024-03-25T18:26:03.396 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.395+0000 7f9213610040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:26:03.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.426+0000 7fde427fc640 1 -- 172.21.15.146:0/265934307 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bb6f7c78-451d-4f40-b363-5de7e3f0fcb9", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3444710244 0 940483276) 0x7fde44005bd0 con 0x7fde4c138720 2024-03-25T18:26:03.426 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.426+0000 7fde427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bb6f7c78-451d-4f40-b363-5de7e3f0fcb9", "id": 1}] 2024-03-25T18:26:03.427 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.426+0000 7fde427fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:03.427 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-03-25T18:26:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 10 librados: watch_flush enter 2024-03-25T18:26:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 10 librados: watch_flush exit 2024-03-25T18:26:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 20 client.4121.objecter shutdown clearing up homeless session... 2024-03-25T18:26:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 10 client.4121.objecter successfully canceled tick 2024-03-25T18:26:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 10 monclient: shutdown 2024-03-25T18:26:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 >> v1:172.21.15.102:6790/0 conn(0x7fde4c138720 legacy=0x7fde4c070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.428 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 shutdown_connections 2024-03-25T18:26:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 >> 172.21.15.146:0/265934307 conn(0x7fde4c06b560 msgr2=0x7fde4c0a94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 shutdown_connections 2024-03-25T18:26:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 1 -- 172.21.15.146:0/265934307 wait complete. 2024-03-25T18:26:03.429 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.427+0000 7fde541af640 1 librados: shutdown 2024-03-25T18:26:03.439 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 97cab3fa-01da-417b-ba9e-12ad9c9792e0 2 2024-03-25T18:26:03.448 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.447+0000 7f9213610040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:26:03.501 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:03.501+0000 7f51dce2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.580+0000 7f067e601640 10 monclient: get_monmap_and_config 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.580+0000 7f067e601640 10 monclient: build_initial_monmap 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.580+0000 7f067e601640 10 monclient: monmap: 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:26:03.581295+0000 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:26:03.581295+0000 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:03.581 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:03.582 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.581+0000 7f067e601640 1 Processor -- start 2024-03-25T18:26:03.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.581+0000 7f067e601640 1 -- start start 2024-03-25T18:26:03.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.581+0000 7f067e601640 10 monclient: init 2024-03-25T18:26:03.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:03.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:03.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 10 monclient(hunting): picked mon.noname-a con 0x7f06780b4700 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:03.583 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 10 monclient(hunting): picked mon.noname-b con 0x7f0678142a20 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:03.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 10 monclient(hunting): picked mon.noname-c con 0x7f06780a96b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:03.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780a9d60 con 0x7f06780b4700 2024-03-25T18:26:03.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067813ff10 con 0x7f06780a96b0 2024-03-25T18:26:03.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780aa1f0 con 0x7f0678142a20 2024-03-25T18:26:03.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:03.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067e601640 10 monclient(hunting): authenticate will time out at 7395.949631s 2024-03-25T18:26:03.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067cb9e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f06780a96b0 0x7f067813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49410/0 (socket says 172.21.15.146:49410) 2024-03-25T18:26:03.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.582+0000 7f067cb9e640 1 -- 172.21.15.146:0/2791642702 learned_addr learned my addr 172.21.15.146:0/2791642702 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:03.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.585+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559271748 0 0) 0x7f06780aa1f0 con 0x7f0678142a20 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.585+0000 7f0676ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.585+0000 7f0676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.585+0000 7f0676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.585+0000 7f0676ffd640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.585+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0660003520 con 0x7f0678142a20 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849524729 0 0) 0x7f06780a9d60 con 0x7f06780b4700 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06780aa1f0 con 0x7f06780b4700 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416985216 0 0) 0x7f067813ff10 con 0x7f06780a96b0 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:26:03.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.586+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06780a9d60 con 0x7f06780a96b0 2024-03-25T18:26:03.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2969002997 0 0) 0x7f0660003520 con 0x7f0678142a20 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 >> v1:172.21.15.146:6789/0 conn(0x7f06780a96b0 legacy=0x7f067813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 >> v1:172.21.15.102:6789/0 conn(0x7f06780b4700 legacy=0x7f0678142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 1 monclient: found mon.noname-b 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780b4ae0 con 0x7f0678142a20 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f066c001690 con 0x7f0678142a20 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f0676ffd640 10 monclient: dump: 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f067e601640 5 monclient: authenticate success, global_id 4124 2024-03-25T18:26:03.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.588+0000 7f067e601640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:03.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f066c0017f0 con 0x7f0678142a20 2024-03-25T18:26:03.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f0676ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:03.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f067e601640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:03.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f0676ffd640 1 -- 172.21.15.146:0/2791642702 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f066c001c00 con 0x7f0678142a20 2024-03-25T18:26:03.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f0676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f067e601640 4 set_mon_vals no callback set 2024-03-25T18:26:03.593 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f0676ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f0676ffd640 10 monclient: dump: 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.590+0000 7f067e601640 10 monclient: shutdown 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.591+0000 7f067e601640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:03.594 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.591+0000 7f067e601640 1 -- 172.21.15.146:0/2791642702 >> v1:172.21.15.102:6790/0 conn(0x7f0678142a20 legacy=0x7f067813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.591+0000 7f067e601640 1 -- 172.21.15.146:0/2791642702 shutdown_connections 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.591+0000 7f067e601640 1 -- 172.21.15.146:0/2791642702 >> 172.21.15.146:0/2791642702 conn(0x7f06780668e0 msgr2=0x7f0678066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.591+0000 7f067e601640 1 -- 172.21.15.146:0/2791642702 shutdown_connections 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.591+0000 7f067e601640 1 -- 172.21.15.146:0/2791642702 wait complete. 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 10 monclient: build_initial_monmap 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 10 monclient: monmap: 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:03.595 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 1 librados: starting msgr at 2024-03-25T18:26:03.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 1 librados: starting objecter 2024-03-25T18:26:03.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 1 Processor -- start 2024-03-25T18:26:03.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 1 -- start start 2024-03-25T18:26:03.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 1 librados: setting wanted keys 2024-03-25T18:26:03.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 1 librados: calling monclient init 2024-03-25T18:26:03.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.592+0000 7f067e601640 10 monclient: init 2024-03-25T18:26:03.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:03.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:03.599 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 10 monclient(hunting): picked mon.noname-c con 0x7f06780a96b0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 10 monclient(hunting): picked mon.noname-b con 0x7f06780b4700 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 10 monclient(hunting): picked mon.noname-a con 0x7f0678142a20 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780ad240 con 0x7f0678142a20 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780adf30 con 0x7f06780b4700 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780ae210 con 0x7f06780a96b0 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.593+0000 7f067e601640 10 monclient(hunting): authenticate will time out at 7395.961225s 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.594+0000 7f06777fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f06780b4700 0x7f06780ae910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49412/0 (socket says 172.21.15.146:49412) 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.594+0000 7f06777fe640 1 -- 172.21.15.146:0/3069767389 learned_addr learned my addr 172.21.15.146:0/3069767389 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.594+0000 7f0674ff9640 10 client.?.objecter ms_handle_connect 0x7f06780b4700 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.594+0000 7f0674ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.594+0000 7f0674ff9640 10 client.?.objecter ms_handle_connect 0x7f06780a96b0 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.594+0000 7f0674ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:03.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.594+0000 7f0674ff9640 10 client.?.objecter ms_handle_connect 0x7f0678142a20 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.594+0000 7f0674ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.596+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521023432 0 0) 0x7f06780ae210 con 0x7f06780a96b0 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.596+0000 7f0674ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.596+0000 7f0674ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.596+0000 7f0674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.596+0000 7f0674ff9640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.596+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0664004eb0 con 0x7f06780a96b0 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606533711 0 0) 0x7f06780ad240 con 0x7f0678142a20 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06780ae210 con 0x7f0678142a20 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056313147 0 0) 0x7f06780adf30 con 0x7f06780b4700 2024-03-25T18:26:03.601 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:26:03.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.597+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06780ad240 con 0x7f06780b4700 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.601+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2570084653 0 0) 0x7f0664004eb0 con 0x7f06780a96b0 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.601+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06780adf30 con 0x7f06780a96b0 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.601+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0668001a10 con 0x7f06780a96b0 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.601+0000 7f0674ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.601+0000 7f0674ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.601+0000 7f0674ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.603 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.604 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.604 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.604 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.601+0000 7f0674ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3751946408 0 0) 0x7f06780ae210 con 0x7f0678142a20 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0664004eb0 con 0x7f0678142a20 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0670001b50 con 0x7f0678142a20 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.606 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964308520 0 0) 0x7f06780ad240 con 0x7f06780b4700 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06780ae210 con 0x7f06780b4700 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f066c001b00 con 0x7f06780b4700 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.607 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 543707016 0 0) 0x7f06780adf30 con 0x7f06780a96b0 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 >> v1:172.21.15.146:6789/0 conn(0x7f06780b4700 legacy=0x7f06780ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 >> v1:172.21.15.102:6789/0 conn(0x7f0678142a20 legacy=0x7f06780ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 monclient: found mon.c 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780ae4f0 con 0x7f06780a96b0 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:03.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f0674ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f067e601640 5 monclient: authenticate success, global_id 4127 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f067e601640 10 monclient: _renew_subs 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f067e601640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.604+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06780abe90 con 0x7f06780a96b0 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 10 monclient: _renew_subs 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06781e3270 con 0x7f06780a96b0 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 1 librados: init done 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.605+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067804f730 con 0x7f06780a96b0 2024-03-25T18:26:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06680035f0 con 0x7f06780a96b0 2024-03-25T18:26:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0668003900 con 0x7f06780a96b0 2024-03-25T18:26:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 10 monclient: dump: 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0668003e30 con 0x7f06780a96b0 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f06767fc640 4 set_mon_vals no callback set 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 565764434 0 0) 0x7f0668005450 con 0x7f06780a96b0 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 10 client.4127.objecter ms_dispatch 0x7f0678000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 3 client.4127.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:26:03.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 3 client.4127.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:26:03.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.608+0000 7f0674ff9640 20 client.4127.objecter dump_active .. 0 homeless 2024-03-25T18:26:03.649 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:03.649+0000 7f51dce2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:26:03.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.703+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0668005140 con 0x7f06780a96b0 2024-03-25T18:26:03.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.703+0000 7f0674ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:03.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.703+0000 7f0674ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:03.769 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:03.768+0000 7f51dce2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:26:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.776+0000 7f067e601640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "97cab3fa-01da-417b-ba9e-12ad9c9792e0", "id": 2}] 2024-03-25T18:26:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.776+0000 7f067e601640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "97cab3fa-01da-417b-ba9e-12ad9c9792e0", "id": 2}] 2024-03-25T18:26:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.776+0000 7f067e601640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:03.776 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.776+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd new", "uuid": "97cab3fa-01da-417b-ba9e-12ad9c9792e0", "id": 2} v 0) v1 -- 0x7f06780b7170 con 0x7f06780a96b0 2024-03-25T18:26:03.815 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:03.815+0000 7f51dce2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:26:03.827 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.827+0000 7f0674ff9640 1 -- 172.21.15.146:0/3069767389 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "97cab3fa-01da-417b-ba9e-12ad9c9792e0", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2788691506 0 216599125) 0x7f0668005680 con 0x7f06780a96b0 2024-03-25T18:26:03.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.827+0000 7f0674ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "97cab3fa-01da-417b-ba9e-12ad9c9792e0", "id": 2}] 2024-03-25T18:26:03.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.827+0000 7f0674ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:03.828 INFO:teuthology.orchestra.run.smithi146.stdout:2 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 10 librados: watch_flush enter 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 10 librados: watch_flush exit 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 20 client.4127.objecter shutdown clearing up homeless session... 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 10 client.4127.objecter successfully canceled tick 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 10 monclient: shutdown 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 >> v1:172.21.15.102:6790/0 conn(0x7f06780a96b0 legacy=0x7f067813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 shutdown_connections 2024-03-25T18:26:03.829 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 >> 172.21.15.146:0/3069767389 conn(0x7f06780668e0 msgr2=0x7f06780b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:03.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 shutdown_connections 2024-03-25T18:26:03.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 1 -- 172.21.15.146:0/3069767389 wait complete. 2024-03-25T18:26:03.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.828+0000 7f067e601640 1 librados: shutdown 2024-03-25T18:26:03.840 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new ad32c648-b51d-4308-9317-5d70afde38f0 3 2024-03-25T18:26:03.847 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.846+0000 7f9213610040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:26:03.934 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:03.934+0000 7f51dce2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:26:03.979 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.979+0000 7f1497723640 10 monclient: get_monmap_and_config 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.979+0000 7f1497723640 10 monclient: build_initial_monmap 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.979+0000 7f1497723640 10 monclient: monmap: 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:26:03.980154+0000 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:26:03.980154+0000 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:03.980 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.980+0000 7f1497723640 1 Processor -- start 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.980+0000 7f1497723640 1 -- start start 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.980+0000 7f1497723640 10 monclient: init 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 10 monclient(hunting): picked mon.noname-b con 0x7f1490070210 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 10 monclient(hunting): picked mon.noname-a con 0x7f14900b06d0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 10 monclient(hunting): picked mon.noname-c con 0x7f14900705f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:03.982 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14900b0b80 con 0x7f14900b06d0 2024-03-25T18:26:03.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14900b0430 con 0x7f14900705f0 2024-03-25T18:26:03.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1490071670 con 0x7f1490070210 2024-03-25T18:26:03.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:03.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1497723640 10 monclient(hunting): authenticate will time out at 7396.348471s 2024-03-25T18:26:03.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1495cc0640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f14900705f0 0x7f149014b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49424/0 (socket says 172.21.15.146:49424) 2024-03-25T18:26:03.983 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.981+0000 7f1495cc0640 1 -- 172.21.15.146:0/4052221933 learned_addr learned my addr 172.21.15.146:0/4052221933 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:03.983 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:03.980+0000 7f9213610040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:26:03.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.983+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938425000 0 0) 0x7f1488001490 con 0x7f14900705f0 2024-03-25T18:26:03.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.983+0000 7f1487fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.983+0000 7f1487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.983+0000 7f1487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.983+0000 7f1487fff640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:26:03.984 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.984+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f147c003520 con 0x7f14900705f0 2024-03-25T18:26:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.984+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248742798 0 0) 0x7f1490071670 con 0x7f1490070210 2024-03-25T18:26:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.984+0000 7f1487fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.984+0000 7f1487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.984+0000 7f1487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.984+0000 7f1487fff640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:26:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.984+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1488001490 con 0x7f1490070210 2024-03-25T18:26:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.985+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960702989 0 0) 0x7f14900b0b80 con 0x7f14900b06d0 2024-03-25T18:26:03.985 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.985+0000 7f1487fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:03.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.985+0000 7f1487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:03.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.985+0000 7f1487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:03.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.985+0000 7f1487fff640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:26:03.986 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.985+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1490071670 con 0x7f14900b06d0 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024909573 0 0) 0x7f147c003520 con 0x7f14900705f0 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 >> v1:172.21.15.102:6790/0 conn(0x7f1490070210 legacy=0x7f1490140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 >> v1:172.21.15.102:6789/0 conn(0x7f14900b06d0 legacy=0x7f14900afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 1 monclient: found mon.noname-c 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490070a60 con 0x7f14900705f0 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1488001a00 con 0x7f14900705f0 2024-03-25T18:26:03.988 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1497723640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1497723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1487fff640 10 monclient: dump: 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:03.989 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:03.990 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:03.987+0000 7f1497723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:04.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.012+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1488001b60 con 0x7f14900705f0 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.012+0000 7f1487fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.012+0000 7f1487fff640 1 -- 172.21.15.146:0/4052221933 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1488001ef0 con 0x7f14900705f0 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.012+0000 7f1487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.012+0000 7f1487fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.012+0000 7f1487fff640 10 monclient: dump: 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.013 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.014 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.014 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.012+0000 7f1497723640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:04.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.012+0000 7f1497723640 4 set_mon_vals no callback set 2024-03-25T18:26:04.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.013+0000 7f1497723640 10 monclient: shutdown 2024-03-25T18:26:04.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.013+0000 7f1497723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:04.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.013+0000 7f1497723640 1 -- 172.21.15.146:0/4052221933 >> v1:172.21.15.146:6789/0 conn(0x7f14900705f0 legacy=0x7f149014b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.013+0000 7f1497723640 1 -- 172.21.15.146:0/4052221933 shutdown_connections 2024-03-25T18:26:04.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.013+0000 7f1497723640 1 -- 172.21.15.146:0/4052221933 >> 172.21.15.146:0/4052221933 conn(0x7f14900b5cd0 msgr2=0x7f14900b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.013+0000 7f1497723640 1 -- 172.21.15.146:0/4052221933 shutdown_connections 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.013+0000 7f1497723640 1 -- 172.21.15.146:0/4052221933 wait complete. 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.014+0000 7f1497723640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.014+0000 7f1497723640 10 monclient: monmap: 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.015+0000 7f1497723640 1 librados: starting msgr at 2024-03-25T18:26:04.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.015+0000 7f1497723640 1 librados: starting objecter 2024-03-25T18:26:04.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.015+0000 7f1497723640 1 Processor -- start 2024-03-25T18:26:04.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.015+0000 7f1497723640 1 -- start start 2024-03-25T18:26:04.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.015+0000 7f1497723640 1 librados: setting wanted keys 2024-03-25T18:26:04.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.015+0000 7f1497723640 1 librados: calling monclient init 2024-03-25T18:26:04.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.015+0000 7f1497723640 10 monclient: init 2024-03-25T18:26:04.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:04.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 10 monclient(hunting): picked mon.noname-c con 0x7f1490070210 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 10 monclient(hunting): picked mon.noname-b con 0x7f14900705f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 10 monclient(hunting): picked mon.noname-a con 0x7f14900b06d0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14901d5240 con 0x7f14900b06d0 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14901d5ec0 con 0x7f14900705f0 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14901d6050 con 0x7f1490070210 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1497723640 10 monclient(hunting): authenticate will time out at 7396.383720s 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1494cbe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f14900705f0 0x7f14901d57b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49428/0 (socket says 172.21.15.146:49428) 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.016+0000 7f1494cbe640 1 -- 172.21.15.146:0/2101499170 learned_addr learned my addr 172.21.15.146:0/2101499170 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.017+0000 7f1485ffb640 10 client.?.objecter ms_handle_connect 0x7f14900705f0 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.017+0000 7f1485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.017+0000 7f1485ffb640 10 client.?.objecter ms_handle_connect 0x7f14900b06d0 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.017+0000 7f1485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.017+0000 7f1485ffb640 10 client.?.objecter ms_handle_connect 0x7f1490070210 2024-03-25T18:26:04.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.017+0000 7f1485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.018+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462728078 0 0) 0x7f14901d5ec0 con 0x7f14900705f0 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.018+0000 7f1485ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.018+0000 7f1485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.018+0000 7f1485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.018+0000 7f1485ffb640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1474004eb0 con 0x7f14900705f0 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936848090 0 0) 0x7f14901d6050 con 0x7f1490070210 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14901d5ec0 con 0x7f1490070210 2024-03-25T18:26:04.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531119126 0 0) 0x7f14901d5240 con 0x7f14900b06d0 2024-03-25T18:26:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:26:04.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.019+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14901d6050 con 0x7f14900b06d0 2024-03-25T18:26:04.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.023+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3645409035 0 0) 0x7f1474004eb0 con 0x7f14900705f0 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.023+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14901d5240 con 0x7f14900705f0 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.023+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1480001970 con 0x7f14900705f0 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.023+0000 7f1485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.023+0000 7f1485ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.023+0000 7f1485ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.025 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.023+0000 7f1485ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.024+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3953436550 0 0) 0x7f14901d5ec0 con 0x7f1490070210 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.024+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1474004eb0 con 0x7f1490070210 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.024+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f148c001970 con 0x7f1490070210 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.024+0000 7f1485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.024+0000 7f1485ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.024+0000 7f1485ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.026 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.027 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.027 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.027 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.027 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018867068 0 0) 0x7f14901d6050 con 0x7f14900b06d0 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14901d5ec0 con 0x7f14900b06d0 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1488001620 con 0x7f14900b06d0 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.029 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3185654193 0 0) 0x7f14901d5240 con 0x7f14900705f0 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 >> v1:172.21.15.102:6790/0 conn(0x7f1490070210 legacy=0x7f14901360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 >> v1:172.21.15.102:6789/0 conn(0x7f14900b06d0 legacy=0x7f14901d8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 1 monclient: found mon.b 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14900af1d0 con 0x7f14900705f0 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1485ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:26:04.030 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 10 monclient: _renew_subs 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14900af440 con 0x7f14900705f0 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 10 monclient: _renew_subs 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f149013ced0 con 0x7f14900705f0 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.027+0000 7f1497723640 1 librados: init done 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.028+0000 7f1497723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.028+0000 7f1497723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.028+0000 7f1497723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.031 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.028+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1490135030 con 0x7f14900705f0 2024-03-25T18:26:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.029+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1480003610 con 0x7f14900705f0 2024-03-25T18:26:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.029+0000 7f1485ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:04.032 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.029+0000 7f14877fe640 4 set_mon_vals no callback set 2024-03-25T18:26:04.035 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:04.035+0000 7f51dce2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1480003a20 con 0x7f14900705f0 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 10 monclient: dump: 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.053 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1480003f90 con 0x7f14900705f0 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3545530318 0 0) 0x7f1480006780 con 0x7f14900705f0 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 10 client.4138.objecter ms_dispatch 0x7f1490000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:26:04.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.052+0000 7f1485ffb640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:26:04.113 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:04.111+0000 7f9213610040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:26:04.158 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:04.157+0000 7f9213610040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:26:04.193 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:04.193+0000 7f51dce2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:26:04.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.221+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1480006400 con 0x7f14900705f0 2024-03-25T18:26:04.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.221+0000 7f1485ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.222 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.221+0000 7f1485ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:04.237 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:04.237+0000 7f51dce2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:26:04.270 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:04.268+0000 7f9213610040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:26:04.293 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.293+0000 7f1497723640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ad32c648-b51d-4308-9317-5d70afde38f0", "id": 3}] 2024-03-25T18:26:04.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.293+0000 7f1497723640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ad32c648-b51d-4308-9317-5d70afde38f0", "id": 3}] 2024-03-25T18:26:04.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.293+0000 7f1497723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.294 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.293+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ad32c648-b51d-4308-9317-5d70afde38f0", "id": 3} v 0) v1 -- 0x7f149004f730 con 0x7f14900705f0 2024-03-25T18:26:04.369 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:04.367+0000 7f9213610040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:26:04.396 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:04.396+0000 7f51dce2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:26:04.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.458+0000 7f1485ffb640 1 -- 172.21.15.146:0/2101499170 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ad32c648-b51d-4308-9317-5d70afde38f0", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 4151257960 0 525046306) 0x7f1480004240 con 0x7f14900705f0 2024-03-25T18:26:04.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.458+0000 7f1485ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ad32c648-b51d-4308-9317-5d70afde38f0", "id": 3}] 2024-03-25T18:26:04.459 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.458+0000 7f1485ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:04.459 INFO:teuthology.orchestra.run.smithi146.stdout:3 2024-03-25T18:26:04.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 10 librados: watch_flush enter 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 10 librados: watch_flush exit 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 10 monclient: shutdown 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 >> v1:172.21.15.146:6789/0 conn(0x7f14900705f0 legacy=0x7f14901d57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 shutdown_connections 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 >> 172.21.15.146:0/2101499170 conn(0x7f14900b5cd0 msgr2=0x7f14900afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 shutdown_connections 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 1 -- 172.21.15.146:0/2101499170 wait complete. 2024-03-25T18:26:04.461 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.459+0000 7f1497723640 1 librados: shutdown 2024-03-25T18:26:04.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 53b511a1-22cc-445a-91fb-e4d7cbd8a021 4 2024-03-25T18:26:04.553 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:04.551+0000 7f9213610040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:26:04.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.569+0000 7f64393e6640 10 monclient: get_monmap_and_config 2024-03-25T18:26:04.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.569+0000 7f64393e6640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.570 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.569+0000 7f64393e6640 10 monclient: monmap: 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:26:04.570247+0000 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:26:04.570247+0000 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.570+0000 7f64393e6640 1 Processor -- start 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.570+0000 7f64393e6640 1 -- start start 2024-03-25T18:26:04.571 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.570+0000 7f64393e6640 10 monclient: init 2024-03-25T18:26:04.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.570+0000 7f64393e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.570+0000 7f64393e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:04.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.570+0000 7f64393e6640 10 monclient(hunting): picked mon.noname-b con 0x7f64340b4700 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:04.572 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64393e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6434142a20 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:04.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64393e6640 10 monclient(hunting): picked mon.noname-c con 0x7f64340a96b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:04.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64393e6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64340a9d60 con 0x7f6434142a20 2024-03-25T18:26:04.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64393e6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643413ff10 con 0x7f64340a96b0 2024-03-25T18:26:04.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64393e6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64340aa1f0 con 0x7f64340b4700 2024-03-25T18:26:04.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64393e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64393e6640 10 monclient(hunting): authenticate will time out at 7396.938384s 2024-03-25T18:26:04.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64337fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f64340a96b0 0x7f643413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49436/0 (socket says 172.21.15.146:49436) 2024-03-25T18:26:04.573 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.571+0000 7f64337fe640 1 -- 172.21.15.146:0/1235360998 learned_addr learned my addr 172.21.15.146:0/1235360998 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:04.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255327768 0 0) 0x7f643413ff10 con 0x7f64340a96b0 2024-03-25T18:26:04.574 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f641c003520 con 0x7f64340a96b0 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451411212 0 0) 0x7f64340a9d60 con 0x7f6434142a20 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.573+0000 7f6431ffb640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:26:04.575 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.574+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643413ff10 con 0x7f6434142a20 2024-03-25T18:26:04.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.575+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522115678 0 0) 0x7f64340aa1f0 con 0x7f64340b4700 2024-03-25T18:26:04.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.575+0000 7f6431ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.575+0000 7f6431ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.575+0000 7f6431ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.575+0000 7f6431ffb640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:26:04.576 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.575+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64340a9d60 con 0x7f64340b4700 2024-03-25T18:26:04.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1342760898 0 0) 0x7f641c003520 con 0x7f64340a96b0 2024-03-25T18:26:04.577 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 >> v1:172.21.15.102:6790/0 conn(0x7f64340b4700 legacy=0x7f6434142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 >> v1:172.21.15.102:6789/0 conn(0x7f6434142a20 legacy=0x7f643413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 1 monclient: found mon.noname-c 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64340b4ae0 con 0x7f64340a96b0 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f64393e6640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f64393e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f642c0019d0 con 0x7f64340a96b0 2024-03-25T18:26:04.578 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f6431ffb640 10 monclient: dump: 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.579 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.577+0000 7f64393e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:04.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.579+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f642c001c20 con 0x7f64340a96b0 2024-03-25T18:26:04.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.579+0000 7f6431ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:04.580 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.579+0000 7f6431ffb640 1 -- 172.21.15.146:0/1235360998 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f642c001fb0 con 0x7f64340a96b0 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.579+0000 7f6431ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.579+0000 7f6431ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.579+0000 7f6431ffb640 10 monclient: dump: 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.579+0000 7f64393e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.579+0000 7f64393e6640 4 set_mon_vals no callback set 2024-03-25T18:26:04.581 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.580+0000 7f64393e6640 10 monclient: shutdown 2024-03-25T18:26:04.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.580+0000 7f64393e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:04.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.580+0000 7f64393e6640 1 -- 172.21.15.146:0/1235360998 >> v1:172.21.15.146:6789/0 conn(0x7f64340a96b0 legacy=0x7f643413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.580+0000 7f64393e6640 1 -- 172.21.15.146:0/1235360998 shutdown_connections 2024-03-25T18:26:04.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.580+0000 7f64393e6640 1 -- 172.21.15.146:0/1235360998 >> 172.21.15.146:0/1235360998 conn(0x7f64340668e0 msgr2=0x7f6434066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:04.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.580+0000 7f64393e6640 1 -- 172.21.15.146:0/1235360998 shutdown_connections 2024-03-25T18:26:04.582 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.580+0000 7f64393e6640 1 -- 172.21.15.146:0/1235360998 wait complete. 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 10 monclient: monmap: 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:04.584 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 1 librados: starting msgr at 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 1 librados: starting objecter 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 1 Processor -- start 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 1 -- start start 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 1 librados: setting wanted keys 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 1 librados: calling monclient init 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.581+0000 7f64393e6640 10 monclient: init 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 10 monclient(hunting): picked mon.noname-a con 0x7f64340a96b0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 10 monclient(hunting): picked mon.noname-b con 0x7f64340b4700 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6434142a20 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:04.585 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643413c070 con 0x7f64340a96b0 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64340ac250 con 0x7f64340b4700 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643413bc50 con 0x7f6434142a20 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.582+0000 7f64393e6640 10 monclient(hunting): authenticate will time out at 7396.950168s 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.583+0000 7f64327fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f64340b4700 0x7f64340ac670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49438/0 (socket says 172.21.15.146:49438) 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.583+0000 7f64327fc640 1 -- 172.21.15.146:0/879946801 learned_addr learned my addr 172.21.15.146:0/879946801 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.583+0000 7f641b7fe640 10 client.?.objecter ms_handle_connect 0x7f64340b4700 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.583+0000 7f641b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.583+0000 7f641b7fe640 10 client.?.objecter ms_handle_connect 0x7f6434142a20 2024-03-25T18:26:04.586 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.583+0000 7f641b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.584+0000 7f641b7fe640 10 client.?.objecter ms_handle_connect 0x7f64340a96b0 2024-03-25T18:26:04.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.584+0000 7f641b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.585+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35546258 0 0) 0x7f64340ac250 con 0x7f64340b4700 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.585+0000 7f641b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.585+0000 7f641b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.585+0000 7f641b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.585+0000 7f641b7fe640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.585+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6420004eb0 con 0x7f64340b4700 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175203004 0 0) 0x7f643413bc50 con 0x7f6434142a20 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64340ac250 con 0x7f6434142a20 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735459988 0 0) 0x7f643413c070 con 0x7f64340a96b0 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:26:04.589 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.586+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643413bc50 con 0x7f64340a96b0 2024-03-25T18:26:04.590 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.590+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 809953546 0 0) 0x7f6420004eb0 con 0x7f64340b4700 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.590+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f643413c070 con 0x7f64340b4700 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.590+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f64240019b0 con 0x7f64340b4700 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.590+0000 7f641b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.590+0000 7f641b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.590+0000 7f641b7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.591 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.590+0000 7f641b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.591+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1223598767 0 0) 0x7f643413bc50 con 0x7f64340a96b0 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.591+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6420004eb0 con 0x7f64340a96b0 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.591+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6428001830 con 0x7f64340a96b0 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.591+0000 7f641b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.591+0000 7f641b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:04.592 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.591+0000 7f641b7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.593 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.593+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 102549330 0 0) 0x7f64340ac250 con 0x7f6434142a20 2024-03-25T18:26:04.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.593+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f643413bc50 con 0x7f6434142a20 2024-03-25T18:26:04.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.593+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f642c001f30 con 0x7f6434142a20 2024-03-25T18:26:04.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.593+0000 7f641b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.593+0000 7f641b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:04.595 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.593+0000 7f641b7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2727704889 0 0) 0x7f643413c070 con 0x7f64340b4700 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 >> v1:172.21.15.102:6790/0 conn(0x7f6434142a20 legacy=0x7f64341332f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 >> v1:172.21.15.102:6789/0 conn(0x7f64340a96b0 legacy=0x7f643413b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.596 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 1 monclient: found mon.b 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643413c500 con 0x7f64340b4700 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f641b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 10 monclient: _renew_subs 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6434139c40 con 0x7f64340b4700 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 10 monclient: _renew_subs 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64340acf60 con 0x7f64340b4700 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.594+0000 7f64393e6640 1 librados: init done 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.595+0000 7f64393e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.597 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.595+0000 7f64393e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.595+0000 7f64393e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.598 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.595+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64340ac0c0 con 0x7f64340b4700 2024-03-25T18:26:04.603 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:04.602+0000 7f9213610040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:26:04.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6424003610 con 0x7f64340b4700 2024-03-25T18:26:04.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f64240039a0 con 0x7f64340b4700 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 10 monclient: dump: 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.693 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6424003f50 con 0x7f64340b4700 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f64317fa640 4 set_mon_vals no callback set 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1294834721 0 0) 0x7f6424006690 con 0x7f64340b4700 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 10 client.4144.objecter ms_dispatch 0x7f6434000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6424006af0 con 0x7f64340b4700 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.691+0000 7f641b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:04.759 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:04.757+0000 7f9213610040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:26:04.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.764+0000 7f64393e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "53b511a1-22cc-445a-91fb-e4d7cbd8a021", "id": 4}] 2024-03-25T18:26:04.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.764+0000 7f64393e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "53b511a1-22cc-445a-91fb-e4d7cbd8a021", "id": 4}] 2024-03-25T18:26:04.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.764+0000 7f64393e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.764+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "53b511a1-22cc-445a-91fb-e4d7cbd8a021", "id": 4} v 0) v1 -- 0x7f643413c6f0 con 0x7f64340b4700 2024-03-25T18:26:04.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.828+0000 7f641b7fe640 1 -- 172.21.15.146:0/879946801 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "53b511a1-22cc-445a-91fb-e4d7cbd8a021", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1963538653 0 1697012583) 0x7f64240042d0 con 0x7f64340b4700 2024-03-25T18:26:04.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.828+0000 7f641b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "53b511a1-22cc-445a-91fb-e4d7cbd8a021", "id": 4}] 2024-03-25T18:26:04.828 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.828+0000 7f641b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:04.829 INFO:teuthology.orchestra.run.smithi146.stdout:4 2024-03-25T18:26:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 10 librados: watch_flush enter 2024-03-25T18:26:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 10 librados: watch_flush exit 2024-03-25T18:26:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:26:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:26:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 10 monclient: shutdown 2024-03-25T18:26:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 >> v1:172.21.15.146:6789/0 conn(0x7f64340b4700 legacy=0x7f64340ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.830 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 shutdown_connections 2024-03-25T18:26:04.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 >> 172.21.15.146:0/879946801 conn(0x7f64340668e0 msgr2=0x7f643413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:04.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 shutdown_connections 2024-03-25T18:26:04.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 1 -- 172.21.15.146:0/879946801 wait complete. 2024-03-25T18:26:04.831 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.829+0000 7f64393e6640 1 librados: shutdown 2024-03-25T18:26:04.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 584db0cd-8ea4-4a16-b14b-12197ed63ee4 5 2024-03-25T18:26:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.939+0000 7f99f7a4a640 10 monclient: get_monmap_and_config 2024-03-25T18:26:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.939+0000 7f99f7a4a640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.939+0000 7f99f7a4a640 10 monclient: monmap: 2024-03-25T18:26:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:26:04.940359+0000 2024-03-25T18:26:04.940 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:26:04.940359+0000 2024-03-25T18:26:04.941 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.941 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.941 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:04.941 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:04.941 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:04.941 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.940+0000 7f99f7a4a640 1 Processor -- start 2024-03-25T18:26:04.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.940+0000 7f99f7a4a640 1 -- start start 2024-03-25T18:26:04.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.940+0000 7f99f7a4a640 10 monclient: init 2024-03-25T18:26:04.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f99f00b4700 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f99f0142a20 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f99f00a96b0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f00a9d60 con 0x7f99f00a96b0 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f013ff10 con 0x7f99f00b4700 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f00aa1f0 con 0x7f99f0142a20 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f7a4a640 10 monclient(hunting): authenticate will time out at 7397.308949s 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f57e6640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f99f00b4700 0x7f99f0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49452/0 (socket says 172.21.15.146:49452) 2024-03-25T18:26:04.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.941+0000 7f99f57e6640 1 -- 172.21.15.146:0/2918602094 learned_addr learned my addr 172.21.15.146:0/2918602094 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214725337 0 0) 0x7f99f013ff10 con 0x7f99f00b4700 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99dc003520 con 0x7f99f00b4700 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688991535 0 0) 0x7f99f00aa1f0 con 0x7f99f0142a20 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f013ff10 con 0x7f99f0142a20 2024-03-25T18:26:04.945 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185199426 0 0) 0x7f99f00a9d60 con 0x7f99f00a96b0 2024-03-25T18:26:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:26:04.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.944+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f00aa1f0 con 0x7f99f00a96b0 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 746887382 0 0) 0x7f99f013ff10 con 0x7f99f0142a20 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 >> v1:172.21.15.146:6789/0 conn(0x7f99f00b4700 legacy=0x7f99f0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 >> v1:172.21.15.102:6789/0 conn(0x7f99f00a96b0 legacy=0x7f99f013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 1 monclient: found mon.noname-b 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00b4ae0 con 0x7f99f0142a20 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.948 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f99ec001670 con 0x7f99f0142a20 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99f7a4a640 5 monclient: authenticate success, global_id 4142 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99f7a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99e7fff640 10 monclient: dump: 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.949 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.950 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.950 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.950 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.947+0000 7f99f7a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:04.952 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99ec0017d0 con 0x7f99f0142a20 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99f7a4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99f7a4a640 4 set_mon_vals no callback set 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99e7fff640 1 -- 172.21.15.146:0/2918602094 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f99ec001be0 con 0x7f99f0142a20 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99e7fff640 10 monclient: dump: 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.953 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99f7a4a640 10 monclient: shutdown 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99f7a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99f7a4a640 1 -- 172.21.15.146:0/2918602094 >> v1:172.21.15.102:6790/0 conn(0x7f99f0142a20 legacy=0x7f99f013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99f7a4a640 1 -- 172.21.15.146:0/2918602094 shutdown_connections 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.949+0000 7f99f7a4a640 1 -- 172.21.15.146:0/2918602094 >> 172.21.15.146:0/2918602094 conn(0x7f99f00668e0 msgr2=0x7f99f0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.950+0000 7f99f7a4a640 1 -- 172.21.15.146:0/2918602094 shutdown_connections 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.950+0000 7f99f7a4a640 1 -- 172.21.15.146:0/2918602094 wait complete. 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.950+0000 7f99f7a4a640 10 monclient: build_initial_monmap 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.950+0000 7f99f7a4a640 10 monclient: monmap: 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:04.954 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.950+0000 7f99f7a4a640 1 librados: starting msgr at 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.950+0000 7f99f7a4a640 1 librados: starting objecter 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 1 Processor -- start 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 1 -- start start 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 1 librados: setting wanted keys 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 1 librados: calling monclient init 2024-03-25T18:26:04.955 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 10 monclient: init 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f99f00a96b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f99f00b4700 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f99f0142a20 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f00ad8d0 con 0x7f99f0142a20 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f00adb20 con 0x7f99f00a96b0 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f01368f0 con 0x7f99f00b4700 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f7a4a640 10 monclient(hunting): authenticate will time out at 7397.318968s 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f57e6640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f99f00a96b0 0x7f99f00ab9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49456/0 (socket says 172.21.15.146:49456) 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.951+0000 7f99f57e6640 1 -- 172.21.15.146:0/658700671 learned_addr learned my addr 172.21.15.146:0/658700671 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.952+0000 7f99e5ffb640 10 client.?.objecter ms_handle_connect 0x7f99f00a96b0 2024-03-25T18:26:04.956 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.952+0000 7f99e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.952+0000 7f99e5ffb640 10 client.?.objecter ms_handle_connect 0x7f99f00b4700 2024-03-25T18:26:04.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.952+0000 7f99e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.952+0000 7f99e5ffb640 10 client.?.objecter ms_handle_connect 0x7f99f0142a20 2024-03-25T18:26:04.957 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.952+0000 7f99e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:04.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.953+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589534105 0 0) 0x7f99f00adb20 con 0x7f99f00a96b0 2024-03-25T18:26:04.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.953+0000 7f99e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.953+0000 7f99e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.953+0000 7f99e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.953+0000 7f99e5ffb640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:26:04.958 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.953+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d4004eb0 con 0x7f99f00a96b0 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093804121 0 0) 0x7f99f01368f0 con 0x7f99f00b4700 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f00adb20 con 0x7f99f00b4700 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801912435 0 0) 0x7f99f00ad8d0 con 0x7f99f0142a20 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.954+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f01368f0 con 0x7f99f0142a20 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.958+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3969817473 0 0) 0x7f99d4004eb0 con 0x7f99f00a96b0 2024-03-25T18:26:04.959 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.958+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99f00ad8d0 con 0x7f99f00a96b0 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f99e0001930 con 0x7f99f00a96b0 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.961 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2729777023 0 0) 0x7f99f01368f0 con 0x7f99f0142a20 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99d4004eb0 con 0x7f99f0142a20 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f99e8001c70 con 0x7f99f0142a20 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.962 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1050138482 0 0) 0x7f99f00adb20 con 0x7f99f00b4700 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99f01368f0 con 0x7f99f00b4700 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f99ec001b20 con 0x7f99f00b4700 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.959+0000 7f99e5ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.963 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:04.964 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:04.964 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:04.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.961+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 969690368 0 0) 0x7f99f00ad8d0 con 0x7f99f00a96b0 2024-03-25T18:26:04.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.961+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 >> v1:172.21.15.102:6790/0 conn(0x7f99f00b4700 legacy=0x7f99f00adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.961+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 >> v1:172.21.15.102:6789/0 conn(0x7f99f0142a20 legacy=0x7f99f01e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:04.965 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.961+0000 7f99e5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.961+0000 7f99e5ffb640 1 monclient: found mon.b 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.961+0000 7f99e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99e5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f00ae650 con 0x7f99f00a96b0 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99e5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99e5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 10 monclient: _renew_subs 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f00ae8c0 con 0x7f99f00a96b0 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 10 monclient: _renew_subs 2024-03-25T18:26:04.966 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f00aca70 con 0x7f99f00a96b0 2024-03-25T18:26:04.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.962+0000 7f99f7a4a640 1 librados: init done 2024-03-25T18:26:04.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.963+0000 7f99f7a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.963+0000 7f99f7a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:04.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.963+0000 7f99f7a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:04.967 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:04.963+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f004f730 con 0x7f99f00a96b0 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99e00047e0 con 0x7f99f00a96b0 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f99e0004950 con 0x7f99f00a96b0 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 10 monclient: dump: 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.063 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f99e0004e40 con 0x7f99f00a96b0 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e77fe640 4 set_mon_vals no callback set 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2105247805 0 0) 0x7f99e0006540 con 0x7f99f00a96b0 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 10 client.4150.objecter ms_dispatch 0x7f99f0000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f99e00069a0 con 0x7f99f00a96b0 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.064 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.062+0000 7f99e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:05.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.133+0000 7f99f7a4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "584db0cd-8ea4-4a16-b14b-12197ed63ee4", "id": 5}] 2024-03-25T18:26:05.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.133+0000 7f99f7a4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "584db0cd-8ea4-4a16-b14b-12197ed63ee4", "id": 5}] 2024-03-25T18:26:05.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.133+0000 7f99f7a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.134 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.133+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "584db0cd-8ea4-4a16-b14b-12197ed63ee4", "id": 5} v 0) v1 -- 0x7f99f00b76a0 con 0x7f99f00a96b0 2024-03-25T18:26:05.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.191+0000 7f99e5ffb640 1 -- 172.21.15.146:0/658700671 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "584db0cd-8ea4-4a16-b14b-12197ed63ee4", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3219065807 0 1988412176) 0x7f99e0006b40 con 0x7f99f00a96b0 2024-03-25T18:26:05.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.191+0000 7f99e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "584db0cd-8ea4-4a16-b14b-12197ed63ee4", "id": 5}] 2024-03-25T18:26:05.192 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.191+0000 7f99e5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:05.192 INFO:teuthology.orchestra.run.smithi146.stdout:5 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.192+0000 7f99f7a4a640 10 librados: watch_flush enter 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.192+0000 7f99f7a4a640 10 librados: watch_flush exit 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.192+0000 7f99f7a4a640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.192+0000 7f99f7a4a640 10 client.4150.objecter successfully canceled tick 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.192+0000 7f99f7a4a640 10 monclient: shutdown 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.192+0000 7f99f7a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.192+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 >> v1:172.21.15.146:6789/0 conn(0x7f99f00a96b0 legacy=0x7f99f00ab9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.192+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 shutdown_connections 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.193+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 >> 172.21.15.146:0/658700671 conn(0x7f99f00668e0 msgr2=0x7f99f0066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.193+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 shutdown_connections 2024-03-25T18:26:05.194 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.193+0000 7f99f7a4a640 1 -- 172.21.15.146:0/658700671 wait complete. 2024-03-25T18:26:05.195 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.193+0000 7f99f7a4a640 1 librados: shutdown 2024-03-25T18:26:05.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new f0686c05-1204-4260-b598-6ac099db8c16 6 2024-03-25T18:26:05.265 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:05.265+0000 7f51dce2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:26:05.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.302+0000 7f4be4660640 10 monclient: get_monmap_and_config 2024-03-25T18:26:05.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.302+0000 7f4be4660640 10 monclient: build_initial_monmap 2024-03-25T18:26:05.303 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.302+0000 7f4be4660640 10 monclient: monmap: 2024-03-25T18:26:05.303 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:26:05.302998+0000 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:26:05.302998+0000 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 1 Processor -- start 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 1 -- start start 2024-03-25T18:26:05.304 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 10 monclient: init 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 10 monclient(hunting): picked mon.noname-b con 0x7f4bdc0b4700 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 10 monclient(hunting): picked mon.noname-c con 0x7f4bdc142a20 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 10 monclient(hunting): picked mon.noname-a con 0x7f4bdc0a96b0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0a9d60 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc13ff10 con 0x7f4bdc142a20 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0aa1f0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:05.306 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.303+0000 7f4be4660640 10 monclient(hunting): authenticate will time out at 7397.671264s 2024-03-25T18:26:05.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.304+0000 7f4be1bfb640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4bdc142a20 0x7f4bdc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49460/0 (socket says 172.21.15.146:49460) 2024-03-25T18:26:05.307 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.304+0000 7f4be1bfb640 1 -- 172.21.15.146:0/3790616613 learned_addr learned my addr 172.21.15.146:0/3790616613 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.306+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609652940 0 0) 0x7f4bdc13ff10 con 0x7f4bdc142a20 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.306+0000 7f4be13fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.306+0000 7f4be13fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.306+0000 7f4be13fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.306+0000 7f4be13fa640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.306+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc8003520 con 0x7f4bdc142a20 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928234307 0 0) 0x7f4bdc0aa1f0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:26:05.308 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc13ff10 con 0x7f4bdc0b4700 2024-03-25T18:26:05.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685596571 0 0) 0x7f4bdc0a9d60 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:26:05.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.307+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc0aa1f0 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.310 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590453283 0 0) 0x7f4bc8003520 con 0x7f4bdc142a20 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 >> v1:172.21.15.102:6790/0 conn(0x7f4bdc0b4700 legacy=0x7f4bdc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 >> v1:172.21.15.102:6789/0 conn(0x7f4bdc0a96b0 legacy=0x7f4bdc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 1 monclient: found mon.noname-c 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0b4ae0 con 0x7f4bdc142a20 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:05.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be4660640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4bcc001690 con 0x7f4bdc142a20 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be13fa640 10 monclient: dump: 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.312 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.313 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.313 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.313 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.313 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.310+0000 7f4be4660640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.311+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bcc0017f0 con 0x7f4bdc142a20 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.311+0000 7f4be13fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.311+0000 7f4be13fa640 1 -- 172.21.15.146:0/3790616613 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4bcc001c00 con 0x7f4bdc142a20 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.311+0000 7f4be13fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.311+0000 7f4be13fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.311+0000 7f4be13fa640 10 monclient: dump: 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.316 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.311+0000 7f4be4660640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.311+0000 7f4be4660640 4 set_mon_vals no callback set 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.312+0000 7f4be4660640 10 monclient: shutdown 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.312+0000 7f4be4660640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.312+0000 7f4be4660640 1 -- 172.21.15.146:0/3790616613 >> v1:172.21.15.146:6789/0 conn(0x7f4bdc142a20 legacy=0x7f4bdc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.312+0000 7f4be4660640 1 -- 172.21.15.146:0/3790616613 shutdown_connections 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.312+0000 7f4be4660640 1 -- 172.21.15.146:0/3790616613 >> 172.21.15.146:0/3790616613 conn(0x7f4bdc0668e0 msgr2=0x7f4bdc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:05.317 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.312+0000 7f4be4660640 1 -- 172.21.15.146:0/3790616613 shutdown_connections 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.312+0000 7f4be4660640 1 -- 172.21.15.146:0/3790616613 wait complete. 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 10 monclient: build_initial_monmap 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 10 monclient: monmap: 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 1 librados: starting msgr at 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 1 librados: starting objecter 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 1 Processor -- start 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 1 -- start start 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 1 librados: setting wanted keys 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 1 librados: calling monclient init 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.313+0000 7f4be4660640 10 monclient: init 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 10 monclient(hunting): picked mon.noname-c con 0x7f4bdc0a96b0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 10 monclient(hunting): picked mon.noname-b con 0x7f4bdc0b4700 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 10 monclient(hunting): picked mon.noname-a con 0x7f4bdc142a20 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0ac1b0 con 0x7f4bdc142a20 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc13a640 con 0x7f4bdc0b4700 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc13c430 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:05.319 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be4660640 10 monclient(hunting): authenticate will time out at 7397.681586s 2024-03-25T18:26:05.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be1bfb640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4bdc0b4700 0x7f4bdc138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49476/0 (socket says 172.21.15.146:49476) 2024-03-25T18:26:05.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4be1bfb640 1 -- 172.21.15.146:0/2493849759 learned_addr learned my addr 172.21.15.146:0/2493849759 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:05.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f4bdc0b4700 2024-03-25T18:26:05.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.314+0000 7f4bd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.315+0000 7f4bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f4bdc0a96b0 2024-03-25T18:26:05.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.315+0000 7f4bd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.316+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181699424 0 0) 0x7f4bdc13a640 con 0x7f4bdc0b4700 2024-03-25T18:26:05.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.316+0000 7f4bd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.316+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.316+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.316+0000 7f4bd2ffd640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:26:05.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.316+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc0004eb0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.316+0000 7f4bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f4bdc142a20 2024-03-25T18:26:05.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.317+0000 7f4bd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.317+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340128185 0 0) 0x7f4bdc13c430 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.317+0000 7f4bd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.317+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.317+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.317+0000 7f4bd2ffd640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.317+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc13a640 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.319+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909806314 0 0) 0x7f4bdc0ac1b0 con 0x7f4bdc142a20 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.319+0000 7f4bd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.319+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.319+0000 7f4bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.319+0000 7f4bd2ffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:26:05.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.319+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bdc13c430 con 0x7f4bdc142a20 2024-03-25T18:26:05.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.321+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3787653360 0 0) 0x7f4bc0004eb0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.321+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bdc0ac1b0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.321+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4bcc001bc0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.321+0000 7f4bd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.321+0000 7f4bd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.321+0000 7f4bd2ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.324 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.321+0000 7f4bd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.322+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693627638 0 0) 0x7f4bdc13a640 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.322+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bc0004eb0 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.322+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4bd8001970 con 0x7f4bdc0a96b0 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.322+0000 7f4bd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.322+0000 7f4bd2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.322+0000 7f4bd2ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.325 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.326 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.326 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.326 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.326 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 161059142 0 0) 0x7f4bdc13c430 con 0x7f4bdc142a20 2024-03-25T18:26:05.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bdc13a640 con 0x7f4bdc142a20 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4bd4001bd0 con 0x7f4bdc142a20 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.328 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1675286712 0 0) 0x7f4bdc0ac1b0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 >> v1:172.21.15.102:6790/0 conn(0x7f4bdc0a96b0 legacy=0x7f4bdc13bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 >> v1:172.21.15.102:6789/0 conn(0x7f4bdc142a20 legacy=0x7f4bdc1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 1 monclient: found mon.b 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc13c5c0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.324+0000 7f4bd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:05.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 10 monclient: _renew_subs 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc13b080 con 0x7f4bdc0b4700 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 10 monclient: _renew_subs 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc0ac720 con 0x7f4bdc0b4700 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 1 librados: init done 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.325+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc13a7d0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bcc0011f0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:05.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4bcc004cb0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 10 monclient: dump: 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4be0bf9640 4 set_mon_vals no callback set 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4bcc005220 con 0x7f4bdc0b4700 2024-03-25T18:26:05.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 97700343 0 0) 0x7f4bcc0069f0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 10 client.4156.objecter ms_dispatch 0x7f4bdc000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:26:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:26:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:26:05.333 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.328+0000 7f4bd2ffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T18:26:05.424 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.424+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4bcc0064b0 con 0x7f4bdc0b4700 2024-03-25T18:26:05.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.424+0000 7f4bd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.425 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.424+0000 7f4bd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:05.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.503+0000 7f4be4660640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f0686c05-1204-4260-b598-6ac099db8c16", "id": 6}] 2024-03-25T18:26:05.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.503+0000 7f4be4660640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f0686c05-1204-4260-b598-6ac099db8c16", "id": 6}] 2024-03-25T18:26:05.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.503+0000 7f4be4660640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.504 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.503+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f0686c05-1204-4260-b598-6ac099db8c16", "id": 6} v 0) v1 -- 0x7f4bdc0aca20 con 0x7f4bdc0b4700 2024-03-25T18:26:05.555 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:05.555+0000 7f51dce2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:26:05.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.559+0000 7f4bd2ffd640 1 -- 172.21.15.146:0/2493849759 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0686c05-1204-4260-b598-6ac099db8c16", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2564473191 0 1113811849) 0x7f4bcc006810 con 0x7f4bdc0b4700 2024-03-25T18:26:05.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4bd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f0686c05-1204-4260-b598-6ac099db8c16", "id": 6}] 2024-03-25T18:26:05.560 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4bd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:05.560 INFO:teuthology.orchestra.run.smithi146.stdout:6 2024-03-25T18:26:05.561 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4be4660640 10 librados: watch_flush enter 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4be4660640 10 librados: watch_flush exit 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4be4660640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4be4660640 10 client.4156.objecter successfully canceled tick 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4be4660640 10 monclient: shutdown 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4be4660640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.560+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 >> v1:172.21.15.146:6789/0 conn(0x7f4bdc0b4700 legacy=0x7f4bdc138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.561+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 shutdown_connections 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.561+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 >> 172.21.15.146:0/2493849759 conn(0x7f4bdc0668e0 msgr2=0x7f4bdc13e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.561+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 shutdown_connections 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.561+0000 7f4be4660640 1 -- 172.21.15.146:0/2493849759 wait complete. 2024-03-25T18:26:05.562 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.561+0000 7f4be4660640 1 librados: shutdown 2024-03-25T18:26:05.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 22087808-2aab-4158-ba49-c99624a3580c 7 2024-03-25T18:26:05.612 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:05.611+0000 7f51dce2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:26:05.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.669+0000 7f2312ab3640 10 monclient: get_monmap_and_config 2024-03-25T18:26:05.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.669+0000 7f2312ab3640 10 monclient: build_initial_monmap 2024-03-25T18:26:05.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.669+0000 7f2312ab3640 10 monclient: monmap: 2024-03-25T18:26:05.670 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:05.670 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.670 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:26:05.670171+0000 2024-03-25T18:26:05.670 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:26:05.670171+0000 2024-03-25T18:26:05.670 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:05.671 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.671 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:05.671 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:05.671 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:05.671 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.670+0000 7f2312ab3640 1 Processor -- start 2024-03-25T18:26:05.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.670+0000 7f2312ab3640 1 -- start start 2024-03-25T18:26:05.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.670+0000 7f2312ab3640 10 monclient: init 2024-03-25T18:26:05.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.670+0000 7f2312ab3640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 10 monclient(hunting): picked mon.noname-b con 0x7f230c0b4700 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 10 monclient(hunting): picked mon.noname-a con 0x7f230c142a20 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 10 monclient(hunting): picked mon.noname-c con 0x7f230c0a96b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c0a9d60 con 0x7f230c142a20 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c13ff10 con 0x7f230c0a96b0 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c0aa1f0 con 0x7f230c0b4700 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2312ab3640 10 monclient(hunting): authenticate will time out at 7398.038399s 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2311050640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f230c0a96b0 0x7f230c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49482/0 (socket says 172.21.15.146:49482) 2024-03-25T18:26:05.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.671+0000 7f2311050640 1 -- 172.21.15.146:0/2367378136 learned_addr learned my addr 172.21.15.146:0/2367378136 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:05.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.673+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802448152 0 0) 0x7f230c0aa1f0 con 0x7f230c0b4700 2024-03-25T18:26:05.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.673+0000 7f230b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.673+0000 7f230b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.673+0000 7f230b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.673+0000 7f230b7fe640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:26:05.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.673+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22fc003520 con 0x7f230c0b4700 2024-03-25T18:26:05.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002993149 0 0) 0x7f230c13ff10 con 0x7f230c0a96b0 2024-03-25T18:26:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:26:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f230c0aa1f0 con 0x7f230c0a96b0 2024-03-25T18:26:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330562110 0 0) 0x7f230c0a9d60 con 0x7f230c142a20 2024-03-25T18:26:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:26:05.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.674+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f230c13ff10 con 0x7f230c142a20 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1512984934 0 0) 0x7f230c0aa1f0 con 0x7f230c0a96b0 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 >> v1:172.21.15.102:6790/0 conn(0x7f230c0b4700 legacy=0x7f230c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 >> v1:172.21.15.102:6789/0 conn(0x7f230c142a20 legacy=0x7f230c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c0b4ae0 con 0x7f230c0a96b0 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:05.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f23040019f0 con 0x7f230c0a96b0 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f230b7fe640 10 monclient: dump: 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f2312ab3640 5 monclient: authenticate success, global_id 4159 2024-03-25T18:26:05.680 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.677+0000 7f2312ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:05.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2304001b50 con 0x7f230c0a96b0 2024-03-25T18:26:05.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f230b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:05.682 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f230b7fe640 1 -- 172.21.15.146:0/2367378136 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2304001f60 con 0x7f230c0a96b0 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 4 set_mon_vals no callback set 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f230b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f230b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f230b7fe640 10 monclient: dump: 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.683 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 10 monclient: shutdown 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 1 -- 172.21.15.146:0/2367378136 >> v1:172.21.15.146:6789/0 conn(0x7f230c0a96b0 legacy=0x7f230c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 1 -- 172.21.15.146:0/2367378136 shutdown_connections 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 1 -- 172.21.15.146:0/2367378136 >> 172.21.15.146:0/2367378136 conn(0x7f230c0668e0 msgr2=0x7f230c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 1 -- 172.21.15.146:0/2367378136 shutdown_connections 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.679+0000 7f2312ab3640 1 -- 172.21.15.146:0/2367378136 wait complete. 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.680+0000 7f2312ab3640 10 monclient: build_initial_monmap 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.680+0000 7f2312ab3640 10 monclient: monmap: 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-03-25T18:26:05.684 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.680+0000 7f2312ab3640 1 librados: starting msgr at 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.680+0000 7f2312ab3640 1 librados: starting objecter 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.680+0000 7f2312ab3640 1 Processor -- start 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 1 -- start start 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 1 librados: setting wanted keys 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 1 librados: calling monclient init 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 10 monclient: init 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 10 monclient(hunting): picked mon.noname-a con 0x7f230c0a96b0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 10 monclient(hunting): picked mon.noname-b con 0x7f230c0b4700 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:05.685 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 10 monclient(hunting): picked mon.noname-c con 0x7f230c142a20 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c13cb60 con 0x7f230c0a96b0 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c0ab880 con 0x7f230c0b4700 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c1d7570 con 0x7f230c142a20 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.681+0000 7f2312ab3640 10 monclient(hunting): authenticate will time out at 7398.049088s 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.682+0000 7f230bfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f230c0b4700 0x7f230c13b430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49490/0 (socket says 172.21.15.146:49490) 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.682+0000 7f230bfff640 1 -- 172.21.15.146:0/3551302236 learned_addr learned my addr 172.21.15.146:0/3551302236 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.682+0000 7f23097fa640 10 client.?.objecter ms_handle_connect 0x7f230c0b4700 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.682+0000 7f23097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.682+0000 7f23097fa640 10 client.?.objecter ms_handle_connect 0x7f230c0a96b0 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.682+0000 7f23097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.682+0000 7f23097fa640 10 client.?.objecter ms_handle_connect 0x7f230c142a20 2024-03-25T18:26:05.686 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.682+0000 7f23097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197334382 0 0) 0x7f230c0ab880 con 0x7f230c0b4700 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f0004eb0 con 0x7f230c0b4700 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363234219 0 0) 0x7f230c1d7570 con 0x7f230c142a20 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:26:05.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.684+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f230c0ab880 con 0x7f230c142a20 2024-03-25T18:26:05.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.685+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555720155 0 0) 0x7f230c13cb60 con 0x7f230c0a96b0 2024-03-25T18:26:05.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.685+0000 7f23097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:05.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.685+0000 7f23097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:05.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.685+0000 7f23097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:05.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.685+0000 7f23097fa640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:26:05.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.685+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f230c1d7570 con 0x7f230c0a96b0 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306222689 0 0) 0x7f22f0004eb0 con 0x7f230c0b4700 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f230c13cb60 con 0x7f230c0b4700 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f22f8001a10 con 0x7f230c0b4700 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 10 monclient(hunting): dump: 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.690 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996474875 0 0) 0x7f230c0ab880 con 0x7f230c142a20 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.688+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f0004eb0 con 0x7f230c142a20 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.689+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2304001e60 con 0x7f230c142a20 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.689+0000 7f23097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.689+0000 7f23097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.689+0000 7f23097fa640 10 monclient(hunting): dump: 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.691 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.692 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.692 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.692 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.692 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.692 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.692 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.692 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.692 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 803888389 0 0) 0x7f230c13cb60 con 0x7f230c0b4700 2024-03-25T18:26:05.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 >> v1:172.21.15.102:6790/0 conn(0x7f230c142a20 legacy=0x7f230c1d45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.693 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 >> v1:172.21.15.102:6789/0 conn(0x7f230c0a96b0 legacy=0x7f230c1d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 1 monclient: found mon.b 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c1d3fd0 con 0x7f230c0b4700 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f23097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 10 monclient: _renew_subs 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230c1d81a0 con 0x7f230c0b4700 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 10 monclient: _renew_subs 2024-03-25T18:26:05.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f230c13bd20 con 0x7f230c0b4700 2024-03-25T18:26:05.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.691+0000 7f2312ab3640 1 librados: init done 2024-03-25T18:26:05.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.692+0000 7f2312ab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.692+0000 7f2312ab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.692+0000 7f2312ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.692+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c1d7700 con 0x7f230c0b4700 2024-03-25T18:26:05.731 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:05.731+0000 7f51dce2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:26:05.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22f8001c70 con 0x7f230c0b4700 2024-03-25T18:26:05.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:05.790 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f22f8003a60 con 0x7f230c0b4700 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 10 monclient: dump: 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 3 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:26:05.791 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f22f8004010 con 0x7f230c0b4700 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3760100404 0 0) 0x7f22f8006920 con 0x7f230c0b4700 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 10 client.4162.objecter ms_dispatch 0x7f230c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f230affd640 4 set_mon_vals no callback set 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.789+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f22f8018e00 con 0x7f230c0b4700 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.790+0000 7f23097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:05.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.790+0000 7f23097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:05.842 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:05.840+0000 7f9213610040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:26:05.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.861+0000 7f2312ab3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "22087808-2aab-4158-ba49-c99624a3580c", "id": 7}] 2024-03-25T18:26:05.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.862+0000 7f2312ab3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "22087808-2aab-4158-ba49-c99624a3580c", "id": 7}] 2024-03-25T18:26:05.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.862+0000 7f2312ab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:05.862 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:05.862+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "22087808-2aab-4158-ba49-c99624a3580c", "id": 7} v 0) v1 -- 0x7f230c04f730 con 0x7f230c0b4700 2024-03-25T18:26:06.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.014+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3083274115 0 0) 0x7f22f80042c0 con 0x7f230c0b4700 2024-03-25T18:26:06.134 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:06.132+0000 7f9213610040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:26:06.189 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:06.188+0000 7f9213610040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:26:06.211 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.211+0000 7f23097fa640 1 -- 172.21.15.146:0/3551302236 <== mon.1 v1:172.21.15.146:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "22087808-2aab-4158-ba49-c99624a3580c", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3021131397 0 1371665406) 0x7f22f8016230 con 0x7f230c0b4700 2024-03-25T18:26:06.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.211+0000 7f23097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "22087808-2aab-4158-ba49-c99624a3580c", "id": 7}] 2024-03-25T18:26:06.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.211+0000 7f23097fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:06.212 INFO:teuthology.orchestra.run.smithi146.stdout:7 2024-03-25T18:26:06.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 10 librados: watch_flush enter 2024-03-25T18:26:06.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 10 librados: watch_flush exit 2024-03-25T18:26:06.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T18:26:06.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 10 client.4162.objecter successfully canceled tick 2024-03-25T18:26:06.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 10 monclient: shutdown 2024-03-25T18:26:06.213 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:06.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 >> v1:172.21.15.146:6789/0 conn(0x7f230c0b4700 legacy=0x7f230c13b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 shutdown_connections 2024-03-25T18:26:06.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 >> 172.21.15.146:0/3551302236 conn(0x7f230c0668e0 msgr2=0x7f230c13f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:06.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 shutdown_connections 2024-03-25T18:26:06.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 1 -- 172.21.15.146:0/3551302236 wait complete. 2024-03-25T18:26:06.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-25T18:26:06.212+0000 7f2312ab3640 1 librados: shutdown 2024-03-25T18:26:06.223 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:06.223 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:26:06.223 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:26:06.226 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:26:06.226 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:06.226 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:26:06.226 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:26:06.228 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:26:06.228 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:06.228 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:26:06.229 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:26:06.231 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:26:06.231 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:06.231 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:26:06.231 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:26:06.233 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:26:06.233 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:06.233 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:26:06.234 DEBUG:teuthology.orchestra.run.smithi146:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:26:06.265 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:26:06.265 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:06.266 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:26:06.266 DEBUG:teuthology.orchestra.run.smithi146:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:26:06.268 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:26:06.268 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:06.268 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:26:06.268 DEBUG:teuthology.orchestra.run.smithi146:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:26:06.270 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:26:06.270 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:26:06.270 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:26:06.270 DEBUG:teuthology.orchestra.run.smithi146:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:26:06.272 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:26:06.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:06.381 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-25T18:26:06.379+0000 7f9213610040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:26:06.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.423+0000 7f4f321e8640 10 monclient: get_monmap_and_config 2024-03-25T18:26:06.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.423+0000 7f4f321e8640 10 monclient: build_initial_monmap 2024-03-25T18:26:06.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.423+0000 7f4f321e8640 10 monclient: monmap: 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:26:06.425406+0000 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:26:06.425406+0000 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:06.426 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.424+0000 7f4f321e8640 1 Processor -- start 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.424+0000 7f4f321e8640 1 -- start start 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.424+0000 7f4f321e8640 10 monclient: init 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 10 monclient(hunting): picked mon.noname-c con 0x7f4f2c0b4700 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 10 monclient(hunting): picked mon.noname-b con 0x7f4f2c142a20 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 10 monclient(hunting): picked mon.noname-a con 0x7f4f2c0a96b0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c0a9d60 con 0x7f4f2c0a96b0 2024-03-25T18:26:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c13ff10 con 0x7f4f2c0b4700 2024-03-25T18:26:06.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c0aa1f0 con 0x7f4f2c142a20 2024-03-25T18:26:06.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:06.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.425+0000 7f4f321e8640 10 monclient(hunting): authenticate will time out at 7397.714689s 2024-03-25T18:26:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.429+0000 7f4f2affd640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4f2c142a20 0x7f4f2c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:39624/0 (socket says 172.21.15.102:39624) 2024-03-25T18:26:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.429+0000 7f4f2affd640 1 -- 172.21.15.102:0/1263725033 learned_addr learned my addr 172.21.15.102:0/1263725033 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:06.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.433+0000 7f4f2a7fc640 1 -- 172.21.15.102:0/1263725033 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115905346 0 0) 0x7f4f2c0aa1f0 con 0x7f4f2c142a20 2024-03-25T18:26:06.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.433+0000 7f4f2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:06.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.433+0000 7f4f2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:06.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.433+0000 7f4f2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:06.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.433+0000 7f4f2a7fc640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:26:06.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.433+0000 7f4f2a7fc640 1 -- 172.21.15.102:0/1263725033 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14003520 con 0x7f4f2c142a20 2024-03-25T18:26:06.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 1 -- 172.21.15.102:0/1263725033 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 35459857 0 0) 0x7f4f14003520 con 0x7f4f2c142a20 2024-03-25T18:26:06.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 1 -- 172.21.15.102:0/1263725033 >> v1:172.21.15.146:6789/0 conn(0x7f4f2c0b4700 legacy=0x7f4f2c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 1 -- 172.21.15.102:0/1263725033 >> v1:172.21.15.102:6789/0 conn(0x7f4f2c0a96b0 legacy=0x7f4f2c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 1 -- 172.21.15.102:0/1263725033 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0b4ae0 con 0x7f4f2c142a20 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.443+0000 7f4f2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.444+0000 7f4f321e8640 5 monclient: authenticate success, global_id 4166 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.444+0000 7f4f321e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.444+0000 7f4f2a7fc640 1 -- 172.21.15.102:0/1263725033 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4f20001690 con 0x7f4f2c142a20 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.444+0000 7f4f2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.444+0000 7f4f2a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:06.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.444+0000 7f4f2a7fc640 10 monclient: dump: 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.444+0000 7f4f321e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:06.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.445+0000 7f4f2a7fc640 1 -- 172.21.15.102:0/1263725033 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f20001920 con 0x7f4f2c142a20 2024-03-25T18:26:06.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.445+0000 7f4f2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:06.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.445+0000 7f4f321e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.445+0000 7f4f321e8640 4 set_mon_vals no callback set 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.446+0000 7f4f321e8640 10 monclient: shutdown 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.446+0000 7f4f321e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.446+0000 7f4f321e8640 1 -- 172.21.15.102:0/1263725033 >> v1:172.21.15.102:6790/0 conn(0x7f4f2c142a20 legacy=0x7f4f2c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.446+0000 7f4f321e8640 1 -- 172.21.15.102:0/1263725033 shutdown_connections 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.446+0000 7f4f321e8640 1 -- 172.21.15.102:0/1263725033 >> 172.21.15.102:0/1263725033 conn(0x7f4f2c0668e0 msgr2=0x7f4f2c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.447+0000 7f4f321e8640 1 -- 172.21.15.102:0/1263725033 shutdown_connections 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.447+0000 7f4f321e8640 1 -- 172.21.15.102:0/1263725033 wait complete. 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 10 monclient: build_initial_monmap 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 10 monclient: monmap: 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 0 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:created 0.000000 2024-03-25T18:26:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 1 librados: starting msgr at 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 1 librados: starting objecter 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 1 Processor -- start 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 1 -- start start 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 1 librados: setting wanted keys 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 1 librados: calling monclient init 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.448+0000 7f4f321e8640 10 monclient: init 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.449+0000 7f4f321e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.449+0000 7f4f321e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.449+0000 7f4f321e8640 10 monclient(hunting): picked mon.noname-c con 0x7f4f2c0a96b0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.449+0000 7f4f321e8640 10 monclient(hunting): picked mon.noname-b con 0x7f4f2c0b4700 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.449+0000 7f4f321e8640 10 monclient(hunting): picked mon.noname-a con 0x7f4f2c142a20 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.449+0000 7f4f321e8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c14cb30 con 0x7f4f2c142a20 2024-03-25T18:26:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.449+0000 7f4f321e8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c14c7c0 con 0x7f4f2c0b4700 2024-03-25T18:26:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.450+0000 7f4f321e8640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c14d050 con 0x7f4f2c0a96b0 2024-03-25T18:26:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.450+0000 7f4f321e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:06.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.450+0000 7f4f321e8640 10 monclient(hunting): authenticate will time out at 7397.738963s 2024-03-25T18:26:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.450+0000 7f4f2bfff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4f2c142a20 0x7f4f2c1e5db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33160/0 (socket says 172.21.15.102:33160) 2024-03-25T18:26:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.450+0000 7f4f2bfff640 1 -- 172.21.15.102:0/889563871 learned_addr learned my addr 172.21.15.102:0/889563871 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.451+0000 7f4f13fff640 10 client.?.objecter ms_handle_connect 0x7f4f2c142a20 2024-03-25T18:26:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.451+0000 7f4f13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.451+0000 7f4f13fff640 10 client.?.objecter ms_handle_connect 0x7f4f2c0b4700 2024-03-25T18:26:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.451+0000 7f4f13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:06.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.484+0000 7f4f13fff640 10 client.?.objecter ms_handle_connect 0x7f4f2c0a96b0 2024-03-25T18:26:06.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.484+0000 7f4f13fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:06.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.570+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581400872 0 0) 0x7f4f2c14cb30 con 0x7f4f2c142a20 2024-03-25T18:26:06.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.570+0000 7f4f13fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:06.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.570+0000 7f4f13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:06.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.570+0000 7f4f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:06.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.570+0000 7f4f13fff640 10 monclient(hunting): my global_id is 4182 2024-03-25T18:26:06.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.570+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f18004eb0 con 0x7f4f2c142a20 2024-03-25T18:26:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.576+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544091256 0 0) 0x7f4f2c14d050 con 0x7f4f2c0a96b0 2024-03-25T18:26:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.576+0000 7f4f13fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.576+0000 7f4f13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.576+0000 7f4f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.576+0000 7f4f13fff640 10 monclient(hunting): my global_id is 4169 2024-03-25T18:26:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.576+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f2c14cb30 con 0x7f4f2c0a96b0 2024-03-25T18:26:06.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.577+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 266484234 0 0) 0x7f4f18004eb0 con 0x7f4f2c142a20 2024-03-25T18:26:06.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.577+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f2c14d050 con 0x7f4f2c142a20 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.582+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4f24001d20 con 0x7f4f2c142a20 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.582+0000 7f4f13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.582+0000 7f4f13fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 10 monclient(hunting): dump: 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:06.586 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2438660187 0 0) 0x7f4f2c14d050 con 0x7f4f2c142a20 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 >> v1:172.21.15.102:6790/0 conn(0x7f4f2c0a96b0 legacy=0x7f4f2c14bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 >> v1:172.21.15.146:6789/0 conn(0x7f4f2c0b4700 legacy=0x7f4f2c1db4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 1 monclient: found mon.a 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c1dbbb0 con 0x7f4f2c142a20 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:06.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 5 monclient: authenticate success, global_id 4182 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 10 monclient: _renew_subs 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c1dbe60 con 0x7f4f2c142a20 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 10 client.4182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 10 monclient: _renew_subs 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c1e76b0 con 0x7f4f2c142a20 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.583+0000 7f4f321e8640 1 librados: init done 2024-03-25T18:26:06.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.585+0000 7f4f321e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:06.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.585+0000 7f4f321e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:06.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.585+0000 7f4f321e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:06.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.585+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c04f730 con 0x7f4f2c142a20 2024-03-25T18:26:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.602+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f24003970 con 0x7f4f2c142a20 2024-03-25T18:26:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.602+0000 7f4f13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.602+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4f24003d40 con 0x7f4f2c142a20 2024-03-25T18:26:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.602+0000 7f4f13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.602+0000 7f4f13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.602+0000 7f4f29ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.602+0000 7f4f13fff640 10 monclient: dump: 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:epoch 1 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:06.605 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:26:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.781+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3083274115 0 0) 0x7f4f24003ea0 con 0x7f4f2c142a20 2024-03-25T18:26:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.787+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f4f24015940 con 0x7f4f2c142a20 2024-03-25T18:26:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.787+0000 7f4f13fff640 10 client.4182.objecter ms_dispatch 0x7f4f2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.787+0000 7f4f13fff640 3 client.4182.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.787+0000 7f4f13fff640 3 client.4182.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.787+0000 7f4f13fff640 20 client.4182.objecter dump_active .. 0 homeless 2024-03-25T18:26:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.787+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4f24004150 con 0x7f4f2c142a20 2024-03-25T18:26:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.787+0000 7f4f13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.787+0000 7f4f13fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:06.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.911+0000 7f4f321e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.911+0000 7f4f321e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.911+0000 7f4f321e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:06.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.911+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f2c14e410 con 0x7f4f2c142a20 2024-03-25T18:26:06.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.989+0000 7f4f13fff640 1 -- 172.21.15.102:0/889563871 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f4f240133e0 con 0x7f4f2c142a20 2024-03-25T18:26:06.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.989+0000 7f4f13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:06.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.989+0000 7f4f13fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:06.991 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-25T18:26:06.992 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.990+0000 7f4f321e8640 10 librados: watch_flush enter 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.990+0000 7f4f321e8640 10 librados: watch_flush exit 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.990+0000 7f4f321e8640 20 client.4182.objecter shutdown clearing up homeless session... 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.990+0000 7f4f321e8640 10 client.4182.objecter successfully canceled tick 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.991+0000 7f4f321e8640 10 monclient: shutdown 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.991+0000 7f4f321e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.991+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 >> v1:172.21.15.102:6789/0 conn(0x7f4f2c142a20 legacy=0x7f4f2c1e5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.991+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 shutdown_connections 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.991+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 >> 172.21.15.102:0/889563871 conn(0x7f4f2c0668e0 msgr2=0x7f4f2c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.991+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 shutdown_connections 2024-03-25T18:26:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.991+0000 7f4f321e8640 1 -- 172.21.15.102:0/889563871 wait complete. 2024-03-25T18:26:06.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-25T18:26:06.991+0000 7f4f321e8640 1 librados: shutdown 2024-03-25T18:26:07.008 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:26:07.009 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:26:07.622 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:07.621+0000 7f517d410640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:26:07.724 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:26:07.723+0000 7f517d410640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:26:07.816 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:07.966 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.965+0000 7ff664c55640 10 monclient: get_monmap_and_config 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.965+0000 7ff664c55640 10 monclient: build_initial_monmap 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.965+0000 7ff664c55640 10 monclient: monmap: 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:26:07.966891+0000 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:26:07.966891+0000 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:07.967 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:07.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.966+0000 7ff664c55640 1 Processor -- start 2024-03-25T18:26:07.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.966+0000 7ff664c55640 1 -- start start 2024-03-25T18:26:07.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.966+0000 7ff664c55640 10 monclient: init 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 10 monclient(hunting): picked mon.noname-c con 0x7ff6600b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 10 monclient(hunting): picked mon.noname-b con 0x7ff6600b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 10 monclient(hunting): picked mon.noname-a con 0x7ff66014be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6600af940 con 0x7ff66014be90 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6600b0a70 con 0x7ff6600b4d60 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6600b0d50 con 0x7ff6600b1730 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.967+0000 7ff664c55640 10 monclient(hunting): authenticate will time out at 7399.256755s 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.968+0000 7ff657fff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff6600b1730 0x7ff6600b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:39636/0 (socket says 172.21.15.102:39636) 2024-03-25T18:26:07.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.968+0000 7ff657fff640 1 -- 172.21.15.102:0/3965106860 learned_addr learned my addr 172.21.15.102:0/3965106860 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369743878 0 0) 0x7ff6600af940 con 0x7ff66014be90 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 10 monclient(hunting): my global_id is 4185 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff64c003520 con 0x7ff66014be90 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260925415 0 0) 0x7ff6600b0d50 con 0x7ff6600b1730 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:07.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.971+0000 7ff65dd9b640 10 monclient(hunting): my global_id is 4172 2024-03-25T18:26:07.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.972+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6600af940 con 0x7ff6600b1730 2024-03-25T18:26:07.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.972+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828006098 0 0) 0x7ff6600b0a70 con 0x7ff6600b4d60 2024-03-25T18:26:07.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.972+0000 7ff65dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:07.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.972+0000 7ff65dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:07.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.972+0000 7ff65dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:07.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.972+0000 7ff65dd9b640 10 monclient(hunting): my global_id is 4183 2024-03-25T18:26:07.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.972+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6600b0d50 con 0x7ff6600b4d60 2024-03-25T18:26:07.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.976+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898348563 0 0) 0x7ff64c003520 con 0x7ff66014be90 2024-03-25T18:26:07.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.976+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 >> v1:172.21.15.102:6790/0 conn(0x7ff6600b1730 legacy=0x7ff6600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:07.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 >> v1:172.21.15.146:6789/0 conn(0x7ff6600b4d60 legacy=0x7ff66013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:07.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:07.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6600b1030 con 0x7ff66014be90 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff658001a10 con 0x7ff66014be90 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff664c55640 5 monclient: authenticate success, global_id 4185 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff664c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff65dd9b640 10 monclient: dump: 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:07.980 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:07.981 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:07.981 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:07.981 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:07.981 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:07.981 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:07.981 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:07.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.977+0000 7ff664c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:07.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.979+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff658001c60 con 0x7ff66014be90 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.980+0000 7ff65dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.980+0000 7ff65dd9b640 1 -- 172.21.15.102:0/3965106860 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff658001ff0 con 0x7ff66014be90 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.980+0000 7ff664c55640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.980+0000 7ff664c55640 4 set_mon_vals no callback set 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.980+0000 7ff65dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.980+0000 7ff664c55640 10 monclient: shutdown 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.980+0000 7ff65dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.981+0000 7ff65dd9b640 10 monclient: dump: 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:07.983 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:07.984 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:07.984 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:07.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.981+0000 7ff664c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:07.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.981+0000 7ff664c55640 1 -- 172.21.15.102:0/3965106860 >> v1:172.21.15.102:6789/0 conn(0x7ff66014be90 legacy=0x7ff66014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:07.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.981+0000 7ff664c55640 1 -- 172.21.15.102:0/3965106860 shutdown_connections 2024-03-25T18:26:07.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.981+0000 7ff664c55640 1 -- 172.21.15.102:0/3965106860 >> 172.21.15.102:0/3965106860 conn(0x7ff6600617a0 msgr2=0x7ff660061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:07.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.981+0000 7ff664c55640 1 -- 172.21.15.102:0/3965106860 shutdown_connections 2024-03-25T18:26:07.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.981+0000 7ff664c55640 1 -- 172.21.15.102:0/3965106860 wait complete. 2024-03-25T18:26:07.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.982+0000 7ff664c55640 10 monclient: build_initial_monmap 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.982+0000 7ff664c55640 10 monclient: monmap: 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.982+0000 7ff664c55640 1 librados: starting msgr at 2024-03-25T18:26:07.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.982+0000 7ff664c55640 1 librados: starting objecter 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.982+0000 7ff664c55640 1 Processor -- start 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.983+0000 7ff664c55640 1 -- start start 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.983+0000 7ff664c55640 1 librados: setting wanted keys 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.983+0000 7ff664c55640 1 librados: calling monclient init 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.983+0000 7ff664c55640 10 monclient: init 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.984+0000 7ff664c55640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.984+0000 7ff664c55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.984+0000 7ff664c55640 10 monclient(hunting): picked mon.noname-a con 0x7ff6600b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:07.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.984+0000 7ff664c55640 10 monclient(hunting): picked mon.noname-b con 0x7ff6600b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.984+0000 7ff664c55640 10 monclient(hunting): picked mon.noname-c con 0x7ff66014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff664c55640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66013b860 con 0x7ff6600b1730 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff664c55640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6601dad00 con 0x7ff6600b4d60 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff664c55640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff660139170 con 0x7ff66014be90 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff664c55640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff664c55640 10 monclient(hunting): authenticate will time out at 7399.273973s 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff65ed9d640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff66014be90 0x7ff660132db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:39640/0 (socket says 172.21.15.102:39640) 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff65e59c640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff6600b1730 0x7ff6601db220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33178/0 (socket says 172.21.15.102:33178) 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff65ed9d640 1 -- 172.21.15.102:0/3002936676 learned_addr learned my addr 172.21.15.102:0/3002936676 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff656ffd640 10 client.?.objecter ms_handle_connect 0x7ff66014be90 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:07.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff656ffd640 10 client.?.objecter ms_handle_connect 0x7ff6600b4d60 2024-03-25T18:26:07.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:07.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff656ffd640 10 client.?.objecter ms_handle_connect 0x7ff6600b1730 2024-03-25T18:26:07.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.985+0000 7ff656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:07.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630812406 0 0) 0x7ff6601dad00 con 0x7ff6600b4d60 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 10 monclient(hunting): my global_id is 4186 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff640004eb0 con 0x7ff6600b4d60 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674167486 0 0) 0x7ff660139170 con 0x7ff66014be90 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 10 monclient(hunting): my global_id is 4175 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.988+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6601dad00 con 0x7ff66014be90 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.989+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498230133 0 0) 0x7ff66013b860 con 0x7ff6600b1730 2024-03-25T18:26:07.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.989+0000 7ff656ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:07.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.989+0000 7ff656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:07.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.989+0000 7ff656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:07.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.989+0000 7ff656ffd640 10 monclient(hunting): my global_id is 4188 2024-03-25T18:26:07.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.989+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff660139170 con 0x7ff6600b1730 2024-03-25T18:26:07.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.995+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3550591231 0 0) 0x7ff640004eb0 con 0x7ff6600b4d60 2024-03-25T18:26:07.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.995+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff66013b860 con 0x7ff6600b4d60 2024-03-25T18:26:07.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 719498285 0 0) 0x7ff6601dad00 con 0x7ff66014be90 2024-03-25T18:26:07.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff640004eb0 con 0x7ff66014be90 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff6480018b0 con 0x7ff6600b4d60 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:07.999 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff658001f70 con 0x7ff66014be90 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.996+0000 7ff656ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:08.000 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:08.001 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.998+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2248484267 0 0) 0x7ff660139170 con 0x7ff6600b1730 2024-03-25T18:26:08.001 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:07.998+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6601dad00 con 0x7ff6600b1730 2024-03-25T18:26:08.001 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.000+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff6500019b0 con 0x7ff6600b1730 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.000+0000 7ff656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.000+0000 7ff656ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.000+0000 7ff656ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:08.002 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1079972838 0 0) 0x7ff66013b860 con 0x7ff6600b4d60 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 >> v1:172.21.15.102:6790/0 conn(0x7ff66014be90 legacy=0x7ff660132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 >> v1:172.21.15.102:6789/0 conn(0x7ff6600b1730 legacy=0x7ff6601db220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 1 monclient: found mon.b 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff660139300 con 0x7ff6600b4d60 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:08.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 5 monclient: authenticate success, global_id 4186 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 10 monclient: _renew_subs 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66013d0f0 con 0x7ff6600b4d60 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 10 monclient: _renew_subs 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6601dbb10 con 0x7ff6600b4d60 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.001+0000 7ff664c55640 1 librados: init done 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.002+0000 7ff664c55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.002+0000 7ff664c55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.002+0000 7ff664c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.002+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6601daf40 con 0x7ff6600b4d60 2024-03-25T18:26:08.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.030+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff648001a10 con 0x7ff6600b4d60 2024-03-25T18:26:08.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.030+0000 7ff656ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:08.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.030+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff6480047e0 con 0x7ff6600b4d60 2024-03-25T18:26:08.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.030+0000 7ff656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:08.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.030+0000 7ff656ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.030+0000 7ff656ffd640 10 monclient: dump: 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:08.032 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:08.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.030+0000 7ff65d59a640 4 set_mon_vals no callback set 2024-03-25T18:26:08.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.031+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3347897148 0 0) 0x7ff64801a740 con 0x7ff6600b4d60 2024-03-25T18:26:08.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.032+0000 7ff65e59c640 10 monclient: get_auth_request con 0x7ff64006df30 auth_method 0 2024-03-25T18:26:08.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.032+0000 7ff656ffd640 10 client.4186.objecter ms_handle_connect 0x7ff64006df30 2024-03-25T18:26:08.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.034+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7ff648054470 con 0x7ff6600b4d60 2024-03-25T18:26:08.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.034+0000 7ff656ffd640 10 client.4186.objecter ms_dispatch 0x7ff660000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:08.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.034+0000 7ff656ffd640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:08.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.034+0000 7ff656ffd640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:08.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.034+0000 7ff656ffd640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-25T18:26:08.425 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.424+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff64804ff80 con 0x7ff6600b4d60 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.424+0000 7ff656ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.424+0000 7ff656ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:08.651 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.649+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7ff64804fa30 con 0x7ff6600b4d60 2024-03-25T18:26:08.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.768+0000 7ff664c55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:08.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.768+0000 7ff664c55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:08.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.768+0000 7ff664c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:08.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.768+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6600b2270 con 0x7ff6600b4d60 2024-03-25T18:26:08.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.842+0000 7ff656ffd640 1 -- 172.21.15.102:0/3002936676 <== mon.1 v1:172.21.15.146:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7ff64807b020 con 0x7ff6600b4d60 2024-03-25T18:26:08.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.842+0000 7ff656ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:08.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.842+0000 7ff656ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:08.844 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:26:08.844 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:08.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 10 librados: watch_flush enter 2024-03-25T18:26:08.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 10 librados: watch_flush exit 2024-03-25T18:26:08.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-25T18:26:08.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 10 client.4186.objecter successfully canceled tick 2024-03-25T18:26:08.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 >> v1:172.21.15.146:6800/3848252714 conn(0x7ff64006df30 legacy=0x7ff6400703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 10 monclient: shutdown 2024-03-25T18:26:08.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:08.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 >> v1:172.21.15.146:6789/0 conn(0x7ff6600b4d60 legacy=0x7ff660139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 shutdown_connections 2024-03-25T18:26:08.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 >> 172.21.15.102:0/3002936676 conn(0x7ff6600617a0 msgr2=0x7ff660074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:08.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 shutdown_connections 2024-03-25T18:26:08.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 1 -- 172.21.15.102:0/3002936676 wait complete. 2024-03-25T18:26:08.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:08.845+0000 7ff664c55640 1 librados: shutdown 2024-03-25T18:26:08.862 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:13.132 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:06.797 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.144 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:06.803 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.195 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:06.861 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.206 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:06.864 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.280 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:06.946 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.289 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:06.947 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.337 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:07.004 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.348 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:07.007 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.354 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:07.037 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.414 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:07.097 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.485 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:07.167 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.513 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:07.196 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.530 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:07.202 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.544 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:07.227 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.573 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:07.255 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.590 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:07.261 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:13.756 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-25T18:26:13.750+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:13.759 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-25T18:26:13.751+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:13.913 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-25T18:26:13.906+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:13.925 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-25T18:26:13.919+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:13.965 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-25T18:26:13.960+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:14.087 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-25T18:26:14.081+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:14.152 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-25T18:26:14.145+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:14.169 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-25T18:26:14.162+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:26:14.209 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:07.867 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.222 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:07.888 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.367 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:08.033 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.382 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:08.040 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.420 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:08.102 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.575 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:08.233 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.595 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:08.277 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.597 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:08.263 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.644 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:08.326 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.648 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:08.320 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.739 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:08.405 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.751 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:08.409 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.789 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:08.472 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.933 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:08.591 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.956 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:08.622 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:14.959 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:08.642 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.017 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:08.699 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.020 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:08.692 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.102 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:08.768 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.125 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:08.783 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.144 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:08.826 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.223 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:08.881 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.261 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:08.927 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.316 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:08.998 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.380 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:09.062 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.385 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:09.056 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.407 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:09.074 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.431 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:09.114 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.435 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:09.093 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.514 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:09.172 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.561 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:09.227 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.607 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:09.290 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.667 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:15.675 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:09.358 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.684 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:09.356 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.703 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:09.369 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.731 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:09.414 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.743 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:09.401 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.757+0000 7fdaaa1f6640 10 monclient: get_monmap_and_config 2024-03-25T18:26:15.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.757+0000 7fdaaa1f6640 10 monclient: build_initial_monmap 2024-03-25T18:26:15.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.757+0000 7fdaaa1f6640 10 monclient: monmap: 2024-03-25T18:26:15.758 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:15.758 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:15.758 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:26:15.758500+0000 2024-03-25T18:26:15.758 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:26:15.758500+0000 2024-03-25T18:26:15.759 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:15.759 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:15.759 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:15.759 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:15.759 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:15.759 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.758+0000 7fdaaa1f6640 1 Processor -- start 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.758+0000 7fdaaa1f6640 1 -- start start 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.758+0000 7fdaaa1f6640 10 monclient: init 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa40a96e0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa40b1b10 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa4146060 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa4140040 con 0x7fdaa4146060 2024-03-25T18:26:15.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa40b1fa0 con 0x7fdaa40a96e0 2024-03-25T18:26:15.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa40b24c0 con 0x7fdaa40b1b10 2024-03-25T18:26:15.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:15.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaaa1f6640 10 monclient(hunting): authenticate will time out at 7407.047158s 2024-03-25T18:26:15.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaa2ffd640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fdaa40b1b10 0x7fdaa4145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:48768/0 (socket says 172.21.15.102:48768) 2024-03-25T18:26:15.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.759+0000 7fdaa2ffd640 1 -- 172.21.15.102:0/2391889359 learned_addr learned my addr 172.21.15.102:0/2391889359 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929756137 0 0) 0x7fdaa40b24c0 con 0x7fdaa40b1b10 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda8c003520 con 0x7fdaa40b1b10 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124422208 0 0) 0x7fdaa4140040 con 0x7fdaa4146060 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:15.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:15.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:26:15.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa40b24c0 con 0x7fdaa4146060 2024-03-25T18:26:15.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309380919 0 0) 0x7fdaa40b1fa0 con 0x7fdaa40a96e0 2024-03-25T18:26:15.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:15.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:15.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:15.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:26:15.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.762+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa4140040 con 0x7fdaa40a96e0 2024-03-25T18:26:15.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77789175 0 0) 0x7fda8c003520 con 0x7fdaa40b1b10 2024-03-25T18:26:15.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 >> v1:172.21.15.146:6789/0 conn(0x7fdaa40a96e0 legacy=0x7fdaa40b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:15.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 >> v1:172.21.15.102:6789/0 conn(0x7fdaa4146060 legacy=0x7fdaa414dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:15.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 1 monclient: found mon.noname-b 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa40b15b0 con 0x7fdaa40b1b10 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaaa1f6640 5 monclient: authenticate success, global_id 4202 2024-03-25T18:26:15.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaaa1f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:15.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fda980017d0 con 0x7fdaa40b1b10 2024-03-25T18:26:15.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:15.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:15.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaa27fc640 10 monclient: dump: 2024-03-25T18:26:15.767 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:15.767 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:15.767 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.765+0000 7fdaaa1f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda98001a20 con 0x7fdaa40b1b10 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaa27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaa27fc640 1 -- 172.21.15.102:0/2391889359 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fda98003610 con 0x7fdaa40b1b10 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 4 set_mon_vals no callback set 2024-03-25T18:26:15.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaa27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaa27fc640 10 monclient: dump: 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:15.769 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 10 monclient: shutdown 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/2391889359 >> v1:172.21.15.102:6790/0 conn(0x7fdaa40b1b10 legacy=0x7fdaa4145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/2391889359 shutdown_connections 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/2391889359 >> 172.21.15.102:0/2391889359 conn(0x7fdaa40b7ea0 msgr2=0x7fdaa40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/2391889359 shutdown_connections 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.767+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/2391889359 wait complete. 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.768+0000 7fdaaa1f6640 10 monclient: build_initial_monmap 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.768+0000 7fdaaa1f6640 10 monclient: monmap: 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:15.772 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.768+0000 7fdaaa1f6640 1 librados: starting msgr at 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.768+0000 7fdaaa1f6640 1 librados: starting objecter 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.768+0000 7fdaaa1f6640 1 Processor -- start 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 1 -- start start 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 1 librados: setting wanted keys 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 1 librados: calling monclient init 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 10 monclient: init 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:15.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa40a96e0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa4146060 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa413a3b0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa413b0a0 con 0x7fdaa40a96e0 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa4138f20 con 0x7fdaa413a3b0 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa41390b0 con 0x7fdaa4146060 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.769+0000 7fdaaa1f6640 10 monclient(hunting): authenticate will time out at 7407.057667s 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.770+0000 7fdaa37fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdaa40a96e0 0x7fdaa4139ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34118/0 (socket says 172.21.15.102:34118) 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.770+0000 7fdaa37fe640 1 -- 172.21.15.102:0/538708860 learned_addr learned my addr 172.21.15.102:0/538708860 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.770+0000 7fda8bfff640 10 client.?.objecter ms_handle_connect 0x7fdaa413a3b0 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.770+0000 7fda8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:15.774 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.770+0000 7fda8bfff640 10 client.?.objecter ms_handle_connect 0x7fdaa4146060 2024-03-25T18:26:15.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.770+0000 7fda8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:15.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.770+0000 7fda8bfff640 10 client.?.objecter ms_handle_connect 0x7fdaa40a96e0 2024-03-25T18:26:15.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.770+0000 7fda8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671198626 0 0) 0x7fdaa4138f20 con 0x7fdaa413a3b0 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda90004eb0 con 0x7fdaa413a3b0 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925065 0 0) 0x7fdaa41390b0 con 0x7fdaa4146060 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:26:15.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.772+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa4138f20 con 0x7fdaa4146060 2024-03-25T18:26:15.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.773+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341617424 0 0) 0x7fdaa413b0a0 con 0x7fdaa40a96e0 2024-03-25T18:26:15.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.773+0000 7fda8bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:15.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.773+0000 7fda8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:15.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.773+0000 7fda8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:15.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.773+0000 7fda8bfff640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:26:15.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.773+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa41390b0 con 0x7fdaa40a96e0 2024-03-25T18:26:15.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.776+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 145695702 0 0) 0x7fda90004eb0 con 0x7fdaa413a3b0 2024-03-25T18:26:15.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.776+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaa413b0a0 con 0x7fdaa413a3b0 2024-03-25T18:26:15.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.776+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fda9c001b10 con 0x7fdaa413a3b0 2024-03-25T18:26:15.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.776+0000 7fda8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.776+0000 7fda8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.776+0000 7fda8bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.776+0000 7fda8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090390250 0 0) 0x7fdaa4138f20 con 0x7fdaa4146060 2024-03-25T18:26:15.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda90004eb0 con 0x7fdaa4146060 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fda980036d0 con 0x7fdaa4146060 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:15.780 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 793557612 0 0) 0x7fdaa41390b0 con 0x7fdaa40a96e0 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaa4138f20 con 0x7fdaa40a96e0 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fda940019b0 con 0x7fdaa40a96e0 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.777+0000 7fda8bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:15.781 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:15.782 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:15.782 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:15.783 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1226766792 0 0) 0x7fdaa413b0a0 con 0x7fdaa413a3b0 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 >> v1:172.21.15.102:6790/0 conn(0x7fdaa4146060 legacy=0x7fdaa4136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 >> v1:172.21.15.102:6789/0 conn(0x7fdaa40a96e0 legacy=0x7fdaa4139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 1 monclient: found mon.b 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4139240 con 0x7fdaa413a3b0 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fda8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 5 monclient: authenticate success, global_id 4216 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 10 monclient: _renew_subs 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:15.784 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa4137650 con 0x7fdaa413a3b0 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 10 monclient: _renew_subs 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa4142a90 con 0x7fdaa413a3b0 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.780+0000 7fdaaa1f6640 1 librados: init done 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.781+0000 7fdaaa1f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.781+0000 7fdaaa1f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.781+0000 7fdaaa1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:15.785 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.781+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa4142000 con 0x7fdaa413a3b0 2024-03-25T18:26:15.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.795+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda9c002000 con 0x7fdaa413a3b0 2024-03-25T18:26:15.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.795+0000 7fda8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.795+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fda9c003970 con 0x7fdaa413a3b0 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.795+0000 7fda8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.795+0000 7fda8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.795+0000 7fda8bfff640 10 monclient: dump: 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:15.796 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:15.797 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:15.797 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:15.797 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:15.797 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:15.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.795+0000 7fdaa1ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:15.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.796+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fda9c019940 con 0x7fdaa413a3b0 2024-03-25T18:26:15.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.796+0000 7fdaa37fe640 10 monclient: get_auth_request con 0x7fda9006e060 auth_method 0 2024-03-25T18:26:15.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.796+0000 7fda8bfff640 10 client.4216.objecter ms_handle_connect 0x7fda9006e060 2024-03-25T18:26:15.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.798+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fda9c053640 con 0x7fdaa413a3b0 2024-03-25T18:26:15.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.798+0000 7fda8bfff640 10 client.4216.objecter ms_dispatch 0x7fdaa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:15.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.798+0000 7fda8bfff640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:15.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.798+0000 7fda8bfff640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:15.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:15.798+0000 7fda8bfff640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T18:26:15.805 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:09.463 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.854 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:09.520 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.894 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:09.577 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.964 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:09.647 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.980 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:09.652 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:15.994 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:09.661 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.022 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:09.705 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.027 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.027+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda9c04ef80 con 0x7fdaa413a3b0 2024-03-25T18:26:16.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.027+0000 7fda8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:16.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.027+0000 7fda8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:16.034 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:09.692 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.093 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:09.751 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.142 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:09.808 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.181 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:09.864 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.196+0000 7fdaaa1f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:16.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.196+0000 7fdaaa1f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:16.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.196+0000 7fdaaa1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:16.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.196+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdaa40a9ac0 con 0x7fdaa413a3b0 2024-03-25T18:26:16.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.211+0000 7fda8bfff640 1 -- 172.21.15.102:0/538708860 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fda9c051080 con 0x7fdaa413a3b0 2024-03-25T18:26:16.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.211+0000 7fda8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:16.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.211+0000 7fda8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:16.212 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:26:16.212 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:16.214 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 10 librados: watch_flush enter 2024-03-25T18:26:16.214 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 10 librados: watch_flush exit 2024-03-25T18:26:16.214 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T18:26:16.214 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 10 client.4216.objecter successfully canceled tick 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 >> v1:172.21.15.146:6800/3848252714 conn(0x7fda9006e060 legacy=0x7fda90070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 10 monclient: shutdown 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 >> v1:172.21.15.146:6789/0 conn(0x7fdaa413a3b0 legacy=0x7fdaa4142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 shutdown_connections 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 >> 172.21.15.102:0/538708860 conn(0x7fdaa40b7ea0 msgr2=0x7fdaa413e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 shutdown_connections 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 1 -- 172.21.15.102:0/538708860 wait complete. 2024-03-25T18:26:16.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:16.213+0000 7fdaaa1f6640 1 librados: shutdown 2024-03-25T18:26:16.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:16.259 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:09.942 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.277 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:09.949 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.285 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:09.951 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.311 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:09.994 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.327 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:09.985 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.355 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:10.038 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.384 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:10.042 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.429 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:10.095 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.475 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:10.157 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.548 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:10.231 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.573 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:10.239 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.589 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:10.261 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.614 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:10.272 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.625 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:10.307 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.675 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:10.333 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.721 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:10.387 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.765 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:10.448 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.835 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:10.518 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.864 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:10.531 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.879 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:10.551 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:16.904 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:10.562 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.057 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:10.740 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.126 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:10.808 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.137 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:10.820 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.176 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:10.848 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.191 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:10.874 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.194 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:10.853 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.249 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:10.907 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.267 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:10.933 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.356 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:11.022 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.401 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:11.067 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.450 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:11.108 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.456 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:11.122 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.549 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:11.207 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.590 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:11.272 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.644 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:11.327 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.667 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:11.349 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.704 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:11.376 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.724 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:11.407 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:17.761 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:11.433 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:22.483 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:16.165 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:22.536 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:16.195 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:22.562 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:16.220 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:22.802 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:16.468 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:22.891 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:16.558 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:22.962 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:16.633 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:23.004 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:16.686 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:23.032 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:23.092 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:16.775 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:23.129 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.128+0000 7f23c93b6640 10 monclient: get_monmap_and_config 2024-03-25T18:26:23.129 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.128+0000 7f23c93b6640 10 monclient: build_initial_monmap 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.128+0000 7f23c93b6640 10 monclient: monmap: 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:26:23.129320+0000 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:26:23.129320+0000 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.129+0000 7f23c93b6640 1 Processor -- start 2024-03-25T18:26:23.130 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.129+0000 7f23c93b6640 1 -- start start 2024-03-25T18:26:23.131 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.129+0000 7f23c93b6640 10 monclient: init 2024-03-25T18:26:23.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:23.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:23.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 10 monclient(hunting): picked mon.noname-a con 0x7f23c40a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:23.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 10 monclient(hunting): picked mon.noname-c con 0x7f23c413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:23.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 10 monclient(hunting): picked mon.noname-b con 0x7f23c41430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:23.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c4142520 con 0x7f23c40a98f0 2024-03-25T18:26:23.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c40b28a0 con 0x7f23c413f710 2024-03-25T18:26:23.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c40b2a30 con 0x7f23c41430c0 2024-03-25T18:26:23.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:23.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.130+0000 7f23c93b6640 10 monclient(hunting): authenticate will time out at 7414.418602s 2024-03-25T18:26:23.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.131+0000 7f23c37fe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f23c41430c0 0x7f23c414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56090/0 (socket says 172.21.15.102:56090) 2024-03-25T18:26:23.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.131+0000 7f23c37fe640 1 -- 172.21.15.102:0/2348942499 learned_addr learned my addr 172.21.15.102:0/2348942499 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:23.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643733500 0 0) 0x7f23c40b2a30 con 0x7f23c41430c0 2024-03-25T18:26:23.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ac003520 con 0x7f23c41430c0 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443518262 0 0) 0x7f23c40b28a0 con 0x7f23c413f710 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:26:23.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.133+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c40b2a30 con 0x7f23c413f710 2024-03-25T18:26:23.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.134+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460287972 0 0) 0x7f23c4142520 con 0x7f23c40a98f0 2024-03-25T18:26:23.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.134+0000 7f23c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.134+0000 7f23c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.134+0000 7f23c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.134+0000 7f23c1ffb640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:26:23.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.135+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c40b28a0 con 0x7f23c40a98f0 2024-03-25T18:26:23.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3409461041 0 0) 0x7f23ac003520 con 0x7f23c41430c0 2024-03-25T18:26:23.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 >> v1:172.21.15.146:6789/0 conn(0x7f23c413f710 legacy=0x7f23c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 >> v1:172.21.15.102:6789/0 conn(0x7f23c40a98f0 legacy=0x7f23c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 1 monclient: found mon.noname-b 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c40b44c0 con 0x7f23c41430c0 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f23bc001a10 con 0x7f23c41430c0 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c1ffb640 10 monclient: dump: 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.141 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c93b6640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.137+0000 7f23c93b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.140+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23bc001b70 con 0x7f23c41430c0 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.140+0000 7f23c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.140+0000 7f23c1ffb640 1 -- 172.21.15.102:0/2348942499 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f23bc001f80 con 0x7f23c41430c0 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.140+0000 7f23c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.140+0000 7f23c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:23.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.140+0000 7f23c1ffb640 10 monclient: dump: 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.140+0000 7f23c93b6640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:23.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.140+0000 7f23c93b6640 4 set_mon_vals no callback set 2024-03-25T18:26:23.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.141+0000 7f23c93b6640 10 monclient: shutdown 2024-03-25T18:26:23.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.141+0000 7f23c93b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:23.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.141+0000 7f23c93b6640 1 -- 172.21.15.102:0/2348942499 >> v1:172.21.15.102:6790/0 conn(0x7f23c41430c0 legacy=0x7f23c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.142+0000 7f23c93b6640 1 -- 172.21.15.102:0/2348942499 shutdown_connections 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.142+0000 7f23c93b6640 1 -- 172.21.15.102:0/2348942499 >> 172.21.15.102:0/2348942499 conn(0x7f23c40b7ea0 msgr2=0x7f23c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.142+0000 7f23c93b6640 1 -- 172.21.15.102:0/2348942499 shutdown_connections 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.142+0000 7f23c93b6640 1 -- 172.21.15.102:0/2348942499 wait complete. 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.143+0000 7f23c93b6640 10 monclient: build_initial_monmap 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.143+0000 7f23c93b6640 10 monclient: monmap: 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:23.147 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.143+0000 7f23c93b6640 1 librados: starting msgr at 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.143+0000 7f23c93b6640 1 librados: starting objecter 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.143+0000 7f23c93b6640 1 Processor -- start 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.144+0000 7f23c93b6640 1 -- start start 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.144+0000 7f23c93b6640 1 librados: setting wanted keys 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.144+0000 7f23c93b6640 1 librados: calling monclient init 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.144+0000 7f23c93b6640 10 monclient: init 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.144+0000 7f23c93b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.144+0000 7f23c93b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.144+0000 7f23c93b6640 10 monclient(hunting): picked mon.noname-b con 0x7f23c40a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c93b6640 10 monclient(hunting): picked mon.noname-a con 0x7f23c413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c93b6640 10 monclient(hunting): picked mon.noname-c con 0x7f23c41430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:23.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c93b6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c40b1a00 con 0x7f23c413f710 2024-03-25T18:26:23.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c93b6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c4070e30 con 0x7f23c40a98f0 2024-03-25T18:26:23.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c93b6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c4071c50 con 0x7f23c41430c0 2024-03-25T18:26:23.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c93b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:23.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c93b6640 10 monclient(hunting): authenticate will time out at 7414.433529s 2024-03-25T18:26:23.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c37fe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f23c41430c0 0x7f23c41e68e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56092/0 (socket says 172.21.15.102:56092) 2024-03-25T18:26:23.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.145+0000 7f23c37fe640 1 -- 172.21.15.102:0/1090246413 learned_addr learned my addr 172.21.15.102:0/1090246413 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:23.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.146+0000 7f23ab7fe640 10 client.?.objecter ms_handle_connect 0x7f23c41430c0 2024-03-25T18:26:23.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.146+0000 7f23ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.146+0000 7f23ab7fe640 10 client.?.objecter ms_handle_connect 0x7f23c40a98f0 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.146+0000 7f23ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.147+0000 7f23ab7fe640 10 client.?.objecter ms_handle_connect 0x7f23c413f710 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.147+0000 7f23ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263817523 0 0) 0x7f23c4070e30 con 0x7f23c40a98f0 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b0004eb0 con 0x7f23c40a98f0 2024-03-25T18:26:23.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833971144 0 0) 0x7f23c4071c50 con 0x7f23c41430c0 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.148+0000 7f23ab7fe640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.149+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4070e30 con 0x7f23c41430c0 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.149+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155955902 0 0) 0x7f23c40b1a00 con 0x7f23c413f710 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.149+0000 7f23ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.149+0000 7f23ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.149+0000 7f23ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.149+0000 7f23ab7fe640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:26:23.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.149+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4071c50 con 0x7f23c413f710 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.154+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3855170013 0 0) 0x7f23b0004eb0 con 0x7f23c40a98f0 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.154+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c40b1a00 con 0x7f23c40a98f0 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.154+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f23b40018f0 con 0x7f23c40a98f0 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.154+0000 7f23ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.154+0000 7f23ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.154+0000 7f23ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.155 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.156 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.156 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:23.156 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:23.156 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:23.156 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:23.156 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:23.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.154+0000 7f23ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.155+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4124377826 0 0) 0x7f23c4070e30 con 0x7f23c41430c0 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.155+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b0004eb0 con 0x7f23c41430c0 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.155+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f23bc001e80 con 0x7f23c41430c0 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.155+0000 7f23ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.155+0000 7f23ab7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.155+0000 7f23ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.157 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.156+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4220685827 0 0) 0x7f23c4071c50 con 0x7f23c413f710 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.156+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c4070e30 con 0x7f23c413f710 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.156+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f23b8001930 con 0x7f23c413f710 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.156+0000 7f23ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.156+0000 7f23ab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.156+0000 7f23ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:23.158 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:23.159 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:23.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3885049443 0 0) 0x7f23c4070e30 con 0x7f23c413f710 2024-03-25T18:26:23.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 >> v1:172.21.15.102:6790/0 conn(0x7f23c41430c0 legacy=0x7f23c41e68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 >> v1:172.21.15.146:6789/0 conn(0x7f23c40a98f0 legacy=0x7f23c41417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 1 monclient: found mon.a 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c4071f30 con 0x7f23c413f710 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23c93b6640 5 monclient: authenticate success, global_id 4224 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23c93b6640 10 monclient: _renew_subs 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23c93b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:23.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c40b0990 con 0x7f23c413f710 2024-03-25T18:26:23.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23c93b6640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:23.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23c93b6640 10 monclient: _renew_subs 2024-03-25T18:26:23.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23c93b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:23.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.159+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c40b1640 con 0x7f23c413f710 2024-03-25T18:26:23.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.160+0000 7f23c93b6640 1 librados: init done 2024-03-25T18:26:23.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.161+0000 7f23c93b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:23.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.161+0000 7f23c93b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:23.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.161+0000 7f23c93b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:23.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.161+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c406fbd0 con 0x7f23c413f710 2024-03-25T18:26:23.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.181+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b80035f0 con 0x7f23c413f710 2024-03-25T18:26:23.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.181+0000 7f23ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:23.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.181+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f23b8003760 con 0x7f23c413f710 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.181+0000 7f23ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.181+0000 7f23ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.181+0000 7f23ab7fe640 10 monclient: dump: 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:23.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.181+0000 7f23c17fa640 4 set_mon_vals no callback set 2024-03-25T18:26:23.183 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.182+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f23b8019730 con 0x7f23c413f710 2024-03-25T18:26:23.183 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.182+0000 7f23c2ffd640 10 monclient: get_auth_request con 0x7f23b006e140 auth_method 0 2024-03-25T18:26:23.183 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.183+0000 7f23ab7fe640 10 client.4224.objecter ms_handle_connect 0x7f23b006e140 2024-03-25T18:26:23.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.187+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f23b80533b0 con 0x7f23c413f710 2024-03-25T18:26:23.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.187+0000 7f23ab7fe640 10 client.4224.objecter ms_dispatch 0x7f23c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:23.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.187+0000 7f23ab7fe640 3 client.4224.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:23.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.187+0000 7f23ab7fe640 3 client.4224.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:23.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.187+0000 7f23ab7fe640 20 client.4224.objecter dump_active .. 0 homeless 2024-03-25T18:26:23.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.448+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23b804eec0 con 0x7f23c413f710 2024-03-25T18:26:23.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.448+0000 7f23ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:23.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.448+0000 7f23ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:23.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.626+0000 7f23c93b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:23.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.626+0000 7f23c93b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:23.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.626+0000 7f23c93b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:26:23.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.626+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23c4074cc0 con 0x7f23c413f710 2024-03-25T18:26:23.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.642+0000 7f23ab7fe640 1 -- 172.21.15.102:0/1090246413 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f23b8050df0 con 0x7f23c413f710 2024-03-25T18:26:23.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.642+0000 7f23ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:23.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.642+0000 7f23ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:23.643 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:26:23.644 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:23.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 10 librados: watch_flush enter 2024-03-25T18:26:23.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 10 librados: watch_flush exit 2024-03-25T18:26:23.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 20 client.4224.objecter shutdown clearing up homeless session... 2024-03-25T18:26:23.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 10 client.4224.objecter successfully canceled tick 2024-03-25T18:26:23.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 >> v1:172.21.15.146:6800/3848252714 conn(0x7f23b006e140 legacy=0x7f23b0070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 10 monclient: shutdown 2024-03-25T18:26:23.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:23.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 >> v1:172.21.15.102:6789/0 conn(0x7f23c413f710 legacy=0x7f23c4070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 shutdown_connections 2024-03-25T18:26:23.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 >> 172.21.15.102:0/1090246413 conn(0x7f23c40b7ea0 msgr2=0x7f23c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:23.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 shutdown_connections 2024-03-25T18:26:23.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 1 -- 172.21.15.102:0/1090246413 wait complete. 2024-03-25T18:26:23.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:23.645+0000 7f23c93b6640 1 librados: shutdown 2024-03-25T18:26:23.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:27.977 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:21.634 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:27.980 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:21.638 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.066 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:21.723 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.069 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:21.727 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.254 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:21.920 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.260 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:21.926 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.392 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:22.058 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.397 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:22.064 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.421 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:22.103 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.426 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:22.108 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.745 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:22.416 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.748 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:22.420 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.843 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:22.525 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:28.846 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:22.529 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:29.047 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:22.729 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:29.053 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:22.736 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:30.466 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.565+0000 7f0c027e2640 10 monclient: get_monmap_and_config 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.565+0000 7f0c027e2640 10 monclient: build_initial_monmap 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.565+0000 7f0c027e2640 10 monclient: monmap: 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:26:30.565888+0000 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:26:30.565888+0000 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:30.566 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:30.567 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:30.568 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.565+0000 7f0c027e2640 1 Processor -- start 2024-03-25T18:26:30.568 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 1 -- start start 2024-03-25T18:26:30.568 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 10 monclient: init 2024-03-25T18:26:30.568 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:30.568 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:30.568 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 10 monclient(hunting): picked mon.noname-a con 0x7f0bfc0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 10 monclient(hunting): picked mon.noname-c con 0x7f0bfc13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 10 monclient(hunting): picked mon.noname-b con 0x7f0bfc1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc142520 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc0b28a0 con 0x7f0bfc13f710 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc0b2a30 con 0x7f0bfc1430c0 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.566+0000 7f0c027e2640 10 monclient(hunting): authenticate will time out at 7421.854810s 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.567+0000 7f0bfbfff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0bfc0a98f0 0x7f0bfc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53414/0 (socket says 172.21.15.102:53414) 2024-03-25T18:26:30.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.567+0000 7f0bfbfff640 1 -- 172.21.15.102:0/477143911 learned_addr learned my addr 172.21.15.102:0/477143911 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:30.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.570+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769191036 0 0) 0x7f0bfc142520 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.570+0000 7f0bfaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:30.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.570+0000 7f0bfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:30.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.570+0000 7f0bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:30.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.570+0000 7f0bfaffd640 10 monclient(hunting): my global_id is 4227 2024-03-25T18:26:30.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.570+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be4003520 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.571+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159413064 0 0) 0x7f0bfc0b28a0 con 0x7f0bfc13f710 2024-03-25T18:26:30.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.571+0000 7f0bfaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:30.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.571+0000 7f0bfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:30.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.571+0000 7f0bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:30.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.571+0000 7f0bfaffd640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:26:30.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.571+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bfc142520 con 0x7f0bfc13f710 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3710833547 0 0) 0x7f0be4003520 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 >> v1:172.21.15.102:6790/0 conn(0x7f0bfc1430c0 legacy=0x7f0bfc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 >> v1:172.21.15.146:6789/0 conn(0x7f0bfc13f710 legacy=0x7f0bfc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 1 monclient: found mon.noname-a 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc0b44c0 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0bec001690 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0bfaffd640 10 monclient: dump: 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0c027e2640 5 monclient: authenticate success, global_id 4227 2024-03-25T18:26:30.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.574+0000 7f0c027e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.576+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bec0017f0 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.576+0000 7f0bfaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.576+0000 7f0bfaffd640 1 -- 172.21.15.102:0/477143911 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0bec001c00 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.576+0000 7f0bfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.576+0000 7f0bfaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.576+0000 7f0bfaffd640 10 monclient: dump: 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:30.577 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:30.578 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:30.578 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:30.578 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:30.578 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.576+0000 7f0c027e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.576+0000 7f0c027e2640 4 set_mon_vals no callback set 2024-03-25T18:26:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.577+0000 7f0c027e2640 10 monclient: shutdown 2024-03-25T18:26:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.577+0000 7f0c027e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.577+0000 7f0c027e2640 1 -- 172.21.15.102:0/477143911 >> v1:172.21.15.102:6789/0 conn(0x7f0bfc0a98f0 legacy=0x7f0bfc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.577+0000 7f0c027e2640 1 -- 172.21.15.102:0/477143911 shutdown_connections 2024-03-25T18:26:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.577+0000 7f0c027e2640 1 -- 172.21.15.102:0/477143911 >> 172.21.15.102:0/477143911 conn(0x7f0bfc0b7ea0 msgr2=0x7f0bfc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.578+0000 7f0c027e2640 1 -- 172.21.15.102:0/477143911 shutdown_connections 2024-03-25T18:26:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.578+0000 7f0c027e2640 1 -- 172.21.15.102:0/477143911 wait complete. 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.579+0000 7f0c027e2640 10 monclient: build_initial_monmap 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.579+0000 7f0c027e2640 10 monclient: monmap: 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:30.580 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:30.582 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.580+0000 7f0c027e2640 1 librados: starting msgr at 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.580+0000 7f0c027e2640 1 librados: starting objecter 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.580+0000 7f0c027e2640 1 Processor -- start 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.580+0000 7f0c027e2640 1 -- start start 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.580+0000 7f0c027e2640 1 librados: setting wanted keys 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.580+0000 7f0c027e2640 1 librados: calling monclient init 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.580+0000 7f0c027e2640 10 monclient: init 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 10 monclient(hunting): picked mon.noname-a con 0x7f0bfc0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 10 monclient(hunting): picked mon.noname-c con 0x7f0bfc13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 10 monclient(hunting): picked mon.noname-b con 0x7f0bfc1430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:30.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc136b30 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc137710 con 0x7f0bfc1430c0 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bfc137280 con 0x7f0bfc13f710 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0c027e2640 10 monclient(hunting): authenticate will time out at 7421.869341s 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0bfbfff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0bfc0a98f0 0x7f0bfc137c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53418/0 (socket says 172.21.15.102:53418) 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.581+0000 7f0bfbfff640 1 -- 172.21.15.102:0/2251636694 learned_addr learned my addr 172.21.15.102:0/2251636694 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.582+0000 7f0bf8ff9640 10 client.?.objecter ms_handle_connect 0x7f0bfc0a98f0 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.582+0000 7f0bf8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.582+0000 7f0bf8ff9640 10 client.?.objecter ms_handle_connect 0x7f0bfc1430c0 2024-03-25T18:26:30.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.582+0000 7f0bf8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:30.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.584+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810840427 0 0) 0x7f0bfc136b30 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.584+0000 7f0bf8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.584+0000 7f0bf8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.584+0000 7f0bf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.584+0000 7f0bf8ff9640 10 monclient(hunting): my global_id is 4230 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.584+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be8004eb0 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.585+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978610415 0 0) 0x7f0bfc137710 con 0x7f0bfc1430c0 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.585+0000 7f0bf8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.585+0000 7f0bf8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.585+0000 7f0bf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.585+0000 7f0bf8ff9640 10 monclient(hunting): my global_id is 4228 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.585+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bfc136b30 con 0x7f0bfc1430c0 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.586+0000 7f0bf8ff9640 10 client.?.objecter ms_handle_connect 0x7f0bfc13f710 2024-03-25T18:26:30.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.586+0000 7f0bf8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:30.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.589+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155562668 0 0) 0x7f0bfc137280 con 0x7f0bfc13f710 2024-03-25T18:26:30.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.589+0000 7f0bf8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:30.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.589+0000 7f0bf8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:30.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.589+0000 7f0bf8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:30.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.589+0000 7f0bf8ff9640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:26:30.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.589+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bfc137710 con 0x7f0bfc13f710 2024-03-25T18:26:30.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.590+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3318880176 0 0) 0x7f0be8004eb0 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.590+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bfc137280 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.590+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0bec001c40 con 0x7f0bfc0a98f0 2024-03-25T18:26:30.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.590+0000 7f0bf8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:30.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.590+0000 7f0bf8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:30.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.590+0000 7f0bf8ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:30.590 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:30.590 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.590+0000 7f0bf8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.591+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821958100 0 0) 0x7f0bfc136b30 con 0x7f0bfc1430c0 2024-03-25T18:26:30.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.591+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0be8004eb0 con 0x7f0bfc1430c0 2024-03-25T18:26:30.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.592+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0bf4001b50 con 0x7f0bfc1430c0 2024-03-25T18:26:30.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.592+0000 7f0bf8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:30.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.592+0000 7f0bf8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:30.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.592+0000 7f0bf8ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:30.592 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:30.592 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:30.592 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.593 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.593 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:30.593 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:30.593 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:30.593 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:30.593 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:30.593 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3446110631 0 0) 0x7f0be8004eb0 con 0x7f0bfc1430c0 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 >> v1:172.21.15.102:6790/0 conn(0x7f0bfc13f710 legacy=0x7f0bfc1d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 >> v1:172.21.15.102:6789/0 conn(0x7f0bfc0a98f0 legacy=0x7f0bfc137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 1 monclient: found mon.b 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc137410 con 0x7f0bfc1430c0 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:30.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0bf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 5 monclient: authenticate success, global_id 4228 2024-03-25T18:26:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 10 monclient: _renew_subs 2024-03-25T18:26:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bfc13c2e0 con 0x7f0bfc1430c0 2024-03-25T18:26:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 10 monclient: _renew_subs 2024-03-25T18:26:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bfc133610 con 0x7f0bfc1430c0 2024-03-25T18:26:30.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.595+0000 7f0c027e2640 1 librados: init done 2024-03-25T18:26:30.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.597+0000 7f0c027e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:30.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.597+0000 7f0c027e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:30.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.597+0000 7f0c027e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:30.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.597+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bfc1378a0 con 0x7f0bfc1430c0 2024-03-25T18:26:30.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.811+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bf4002020 con 0x7f0bfc1430c0 2024-03-25T18:26:30.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.812+0000 7f0bf8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:30.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.812+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0bf4003b10 con 0x7f0bfc1430c0 2024-03-25T18:26:30.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.812+0000 7f0bf8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.812+0000 7f0bf8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.812+0000 7f0bf8ff9640 10 monclient: dump: 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:30.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.812+0000 7f0bfa7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:30.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.995+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f0bf4019ae0 con 0x7f0bfc1430c0 2024-03-25T18:26:30.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.995+0000 7f0bfbfff640 10 monclient: get_auth_request con 0x7f0be806df20 auth_method 0 2024-03-25T18:26:30.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.996+0000 7f0bf8ff9640 10 client.4228.objecter ms_handle_connect 0x7f0be806df20 2024-03-25T18:26:30.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.998+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f0bf4053850 con 0x7f0bfc1430c0 2024-03-25T18:26:30.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.998+0000 7f0bf8ff9640 10 client.4228.objecter ms_dispatch 0x7f0bfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:30.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.998+0000 7f0bf8ff9640 3 client.4228.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:30.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.998+0000 7f0bf8ff9640 3 client.4228.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:30.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:30.998+0000 7f0bf8ff9640 20 client.4228.objecter dump_active .. 0 homeless 2024-03-25T18:26:31.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.003+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bf404f190 con 0x7f0bfc1430c0 2024-03-25T18:26:31.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.003+0000 7f0bf8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:31.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.003+0000 7f0bf8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:31.260 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.259+0000 7f0c027e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:31.260 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.259+0000 7f0c027e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:31.260 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.259+0000 7f0c027e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:31.260 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.259+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bfc0627b0 con 0x7f0bfc1430c0 2024-03-25T18:26:31.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.283+0000 7f0bf8ff9640 1 -- 172.21.15.102:0/2251636694 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f0bf4051290 con 0x7f0bfc1430c0 2024-03-25T18:26:31.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.283+0000 7f0bf8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:31.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.283+0000 7f0bf8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:31.284 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:26:31.284 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:31.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 10 librados: watch_flush enter 2024-03-25T18:26:31.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 10 librados: watch_flush exit 2024-03-25T18:26:31.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 20 client.4228.objecter shutdown clearing up homeless session... 2024-03-25T18:26:31.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 10 client.4228.objecter successfully canceled tick 2024-03-25T18:26:31.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 >> v1:172.21.15.146:6800/3848252714 conn(0x7f0be806df20 legacy=0x7f0be80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 10 monclient: shutdown 2024-03-25T18:26:31.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:31.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 >> v1:172.21.15.146:6789/0 conn(0x7f0bfc1430c0 legacy=0x7f0bfc132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 shutdown_connections 2024-03-25T18:26:31.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 >> 172.21.15.102:0/2251636694 conn(0x7f0bfc0b7ea0 msgr2=0x7f0bfc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:31.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 shutdown_connections 2024-03-25T18:26:31.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.285+0000 7f0c027e2640 1 -- 172.21.15.102:0/2251636694 wait complete. 2024-03-25T18:26:31.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:31.286+0000 7f0c027e2640 1 librados: shutdown 2024-03-25T18:26:31.296 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:32.430 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:00:26.095 35222== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:32.511 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:00:26.169 35223== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:32.603 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:00:26.261 35224== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:32.677 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:00:26.359 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:32.771 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:00:26.437 35221== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:33.411 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:00:27.093 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:33.442 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:00:27.124 35532== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:33.586 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:00:27.258 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:26:38.103 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:38.247 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.246+0000 7f823be25640 10 monclient: get_monmap_and_config 2024-03-25T18:26:38.247 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.246+0000 7f823be25640 10 monclient: build_initial_monmap 2024-03-25T18:26:38.247 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.246+0000 7f823be25640 10 monclient: monmap: 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:26:38.247547+0000 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:26:38.247547+0000 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:38.248 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.247+0000 7f823be25640 1 Processor -- start 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.248+0000 7f823be25640 1 -- start start 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.248+0000 7f823be25640 10 monclient: init 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.248+0000 7f823be25640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.248+0000 7f823be25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.249+0000 7f823be25640 10 monclient(hunting): picked mon.noname-c con 0x7f82340b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.249+0000 7f823be25640 10 monclient(hunting): picked mon.noname-b con 0x7f82340b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.249+0000 7f823be25640 10 monclient(hunting): picked mon.noname-a con 0x7f823414be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.249+0000 7f823be25640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82340af940 con 0x7f823414be90 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.249+0000 7f823be25640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82340b0a70 con 0x7f82340b4d60 2024-03-25T18:26:38.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.249+0000 7f823be25640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82340b0d50 con 0x7f82340b1730 2024-03-25T18:26:38.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.249+0000 7f823be25640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:38.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.249+0000 7f823be25640 10 monclient(hunting): authenticate will time out at 7429.537011s 2024-03-25T18:26:38.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.255+0000 7f8239bc1640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f82340b4d60 0x7f823413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:59292/0 (socket says 172.21.15.102:59292) 2024-03-25T18:26:38.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.255+0000 7f8239bc1640 1 -- 172.21.15.102:0/2995940030 learned_addr learned my addr 172.21.15.102:0/2995940030 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:38.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.280+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224368521 0 0) 0x7f82340b0a70 con 0x7f82340b4d60 2024-03-25T18:26:38.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.280+0000 7f8238bbf640 10 monclient(hunting): _negotiate 2024-03-25T18:26:38.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.280+0000 7f8238bbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:38.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.280+0000 7f8238bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:38.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.280+0000 7f8238bbf640 10 monclient(hunting): my global_id is 4231 2024-03-25T18:26:38.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.280+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8220003520 con 0x7f82340b4d60 2024-03-25T18:26:38.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.281+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205806266 0 0) 0x7f82340b0d50 con 0x7f82340b1730 2024-03-25T18:26:38.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.281+0000 7f8238bbf640 10 monclient(hunting): _negotiate 2024-03-25T18:26:38.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.281+0000 7f8238bbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:38.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.281+0000 7f8238bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:38.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.281+0000 7f8238bbf640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:26:38.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.281+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82340b0a70 con 0x7f82340b1730 2024-03-25T18:26:38.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.284+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3921294019 0 0) 0x7f8220003520 con 0x7f82340b4d60 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.284+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 >> v1:172.21.15.102:6790/0 conn(0x7f82340b1730 legacy=0x7f82340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.284+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 >> v1:172.21.15.102:6789/0 conn(0x7f823414be90 legacy=0x7f823414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.284+0000 7f8238bbf640 10 monclient: _finish_hunting 0 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.284+0000 7f8238bbf640 1 monclient: found mon.noname-c 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.284+0000 7f8238bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.284+0000 7f8238bbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.284+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82340b1030 con 0x7f82340b4d60 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f8238bbf640 10 monclient: _finish_auth 0 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f8238bbf640 10 monclient: _check_auth_tickets 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f8238bbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8230001690 con 0x7f82340b4d60 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f8238bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:38.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f8238bbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f8238bbf640 10 monclient: dump: 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:38.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f823be25640 5 monclient: authenticate success, global_id 4231 2024-03-25T18:26:38.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.285+0000 7f823be25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:38.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.287+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82300017f0 con 0x7f82340b4d60 2024-03-25T18:26:38.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.287+0000 7f8238bbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:38.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.287+0000 7f8238bbf640 1 -- 172.21.15.102:0/2995940030 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8230001c00 con 0x7f82340b4d60 2024-03-25T18:26:38.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.287+0000 7f8238bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:38.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.287+0000 7f8238bbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:38.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.287+0000 7f8238bbf640 10 monclient: dump: 2024-03-25T18:26:38.291 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.287+0000 7f823be25640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.287+0000 7f823be25640 4 set_mon_vals no callback set 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.288+0000 7f823be25640 10 monclient: shutdown 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.288+0000 7f823be25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.288+0000 7f823be25640 1 -- 172.21.15.102:0/2995940030 >> v1:172.21.15.146:6789/0 conn(0x7f82340b4d60 legacy=0x7f823413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.288+0000 7f823be25640 1 -- 172.21.15.102:0/2995940030 shutdown_connections 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.288+0000 7f823be25640 1 -- 172.21.15.102:0/2995940030 >> 172.21.15.102:0/2995940030 conn(0x7f82340617a0 msgr2=0x7f8234061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.288+0000 7f823be25640 1 -- 172.21.15.102:0/2995940030 shutdown_connections 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.288+0000 7f823be25640 1 -- 172.21.15.102:0/2995940030 wait complete. 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.289+0000 7f823be25640 10 monclient: build_initial_monmap 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.289+0000 7f823be25640 10 monclient: monmap: 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:38.293 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:38.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.290+0000 7f823be25640 1 librados: starting msgr at 2024-03-25T18:26:38.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.290+0000 7f823be25640 1 librados: starting objecter 2024-03-25T18:26:38.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.290+0000 7f823be25640 1 Processor -- start 2024-03-25T18:26:38.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.290+0000 7f823be25640 1 -- start start 2024-03-25T18:26:38.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.290+0000 7f823be25640 1 librados: setting wanted keys 2024-03-25T18:26:38.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.290+0000 7f823be25640 1 librados: calling monclient init 2024-03-25T18:26:38.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.290+0000 7f823be25640 10 monclient: init 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.291+0000 7f823be25640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.291+0000 7f823be25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f823be25640 10 monclient(hunting): picked mon.noname-c con 0x7f82340b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f823be25640 10 monclient(hunting): picked mon.noname-b con 0x7f82340b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f823be25640 10 monclient(hunting): picked mon.noname-a con 0x7f823414be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f823be25640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823413a820 con 0x7f823414be90 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f823be25640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f823413ab00 con 0x7f82340b4d60 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f823be25640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8234139000 con 0x7f82340b1730 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f823be25640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f8239bc1640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f82340b1730 0x7f8234142f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:36810/0 (socket says 172.21.15.102:36810) 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f8239bc1640 1 -- 172.21.15.102:0/2959944458 learned_addr learned my addr 172.21.15.102:0/2959944458 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.292+0000 7f823be25640 10 monclient(hunting): authenticate will time out at 7429.580858s 2024-03-25T18:26:38.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.293+0000 7f822a7fc640 10 client.?.objecter ms_handle_connect 0x7f82340b1730 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.293+0000 7f822a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.293+0000 7f822a7fc640 10 client.?.objecter ms_handle_connect 0x7f82340b4d60 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.293+0000 7f822a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.293+0000 7f822a7fc640 10 client.?.objecter ms_handle_connect 0x7f823414be90 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.293+0000 7f822a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.295+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529244633 0 0) 0x7f823413ab00 con 0x7f82340b4d60 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.295+0000 7f822a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.295+0000 7f822a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.295+0000 7f822a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.295+0000 7f822a7fc640 10 monclient(hunting): my global_id is 4234 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.295+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8218004eb0 con 0x7f82340b4d60 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.296+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203578647 0 0) 0x7f8234139000 con 0x7f82340b1730 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.296+0000 7f822a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.296+0000 7f822a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:38.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.296+0000 7f822a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:38.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.296+0000 7f822a7fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:26:38.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.296+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823413ab00 con 0x7f82340b1730 2024-03-25T18:26:38.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.297+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989007888 0 0) 0x7f823413a820 con 0x7f823414be90 2024-03-25T18:26:38.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.297+0000 7f822a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:38.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.297+0000 7f822a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:38.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.297+0000 7f822a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:38.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.297+0000 7f822a7fc640 10 monclient(hunting): my global_id is 4236 2024-03-25T18:26:38.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.297+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8234139000 con 0x7f823414be90 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.301+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3083395266 0 0) 0x7f8218004eb0 con 0x7f82340b4d60 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.301+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f823413a820 con 0x7f82340b4d60 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.301+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8224001930 con 0x7f82340b4d60 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.301+0000 7f822a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.302+0000 7f822a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.302+0000 7f822a7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:38.303 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:38.304 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:38.304 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:38.304 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:38.304 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:38.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.302+0000 7f822a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:38.308 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.304+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 785111737 0 0) 0x7f823413ab00 con 0x7f82340b1730 2024-03-25T18:26:38.308 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.304+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8218004eb0 con 0x7f82340b1730 2024-03-25T18:26:38.308 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.304+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8230001b40 con 0x7f82340b1730 2024-03-25T18:26:38.308 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.304+0000 7f822a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:38.308 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.304+0000 7f822a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:38.308 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.304+0000 7f822a7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.304+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927024742 0 0) 0x7f8234139000 con 0x7f823414be90 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.304+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f823413ab00 con 0x7f823414be90 2024-03-25T18:26:38.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f822c001ad0 con 0x7f823414be90 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 10 monclient(hunting): dump: 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:38.310 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3445156687 0 0) 0x7f823413a820 con 0x7f82340b4d60 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 >> v1:172.21.15.102:6790/0 conn(0x7f82340b1730 legacy=0x7f8234142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 >> v1:172.21.15.102:6789/0 conn(0x7f823414be90 legacy=0x7f823413b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 1 monclient: found mon.b 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8234139190 con 0x7f82340b4d60 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.305+0000 7f822a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 5 monclient: authenticate success, global_id 4234 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 10 monclient: _renew_subs 2024-03-25T18:26:38.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823413af20 con 0x7f82340b4d60 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 10 monclient: _renew_subs 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8234134f40 con 0x7f82340b4d60 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.306+0000 7f823be25640 1 librados: init done 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.307+0000 7f823be25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.307+0000 7f823be25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.307+0000 7f823be25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:38.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.307+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82340afbd0 con 0x7f82340b4d60 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.320+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82240035f0 con 0x7f82340b4d60 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.320+0000 7f822a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.320+0000 7f822bfff640 4 set_mon_vals no callback set 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.320+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8224003900 con 0x7f82340b4d60 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.320+0000 7f822a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.321+0000 7f822a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.321+0000 7f822a7fc640 10 monclient: dump: 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:38.322 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:38.323 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:38.323 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:38.323 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:38.323 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:38.323 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:38.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.322+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f8224019910 con 0x7f82340b4d60 2024-03-25T18:26:38.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.323+0000 7f8239bc1640 10 monclient: get_auth_request con 0x7f821806e090 auth_method 0 2024-03-25T18:26:38.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.323+0000 7f822a7fc640 10 client.4234.objecter ms_handle_connect 0x7f821806e090 2024-03-25T18:26:38.607 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.607+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f8224053620 con 0x7f82340b4d60 2024-03-25T18:26:38.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.607+0000 7f822a7fc640 10 client.4234.objecter ms_dispatch 0x7f8234000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:38.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.607+0000 7f822a7fc640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:38.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.607+0000 7f822a7fc640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:38.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.607+0000 7f822a7fc640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-25T18:26:38.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.607+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f822404ef60 con 0x7f82340b4d60 2024-03-25T18:26:38.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.607+0000 7f822a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:38.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.607+0000 7f822a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:38.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.857+0000 7f823be25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:38.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.857+0000 7f823be25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:38.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.857+0000 7f823be25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:38.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.857+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82340b25c0 con 0x7f82340b4d60 2024-03-25T18:26:38.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.881+0000 7f822a7fc640 1 -- 172.21.15.102:0/2959944458 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f8224051060 con 0x7f82340b4d60 2024-03-25T18:26:38.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.881+0000 7f822a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:38.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.881+0000 7f822a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:38.882 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:26:38.882 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:38.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.883+0000 7f823be25640 10 librados: watch_flush enter 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 10 librados: watch_flush exit 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 10 client.4234.objecter successfully canceled tick 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 >> v1:172.21.15.146:6800/3848252714 conn(0x7f821806e090 legacy=0x7f8218070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 10 monclient: shutdown 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 >> v1:172.21.15.146:6789/0 conn(0x7f82340b4d60 legacy=0x7f8234134650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 shutdown_connections 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 >> 172.21.15.102:0/2959944458 conn(0x7f82340617a0 msgr2=0x7f82340b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 shutdown_connections 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 1 -- 172.21.15.102:0/2959944458 wait complete. 2024-03-25T18:26:38.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:38.884+0000 7f823be25640 1 librados: shutdown 2024-03-25T18:26:38.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:40.904 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-25T18:26:40.899+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:26:41.038 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-25T18:26:41.034+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:26:41.152 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-25T18:26:41.148+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:26:41.292 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-25T18:26:41.287+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:26:41.587 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-25T18:26:41.583+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:26:41.993 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-25T18:26:41.987+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:26:42.076 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-25T18:26:42.072+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:26:42.268 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-25T18:26:42.265+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:26:45.706 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:45.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.821+0000 7ff73d498640 10 monclient: get_monmap_and_config 2024-03-25T18:26:45.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.821+0000 7ff73d498640 10 monclient: build_initial_monmap 2024-03-25T18:26:45.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.821+0000 7ff73d498640 10 monclient: monmap: 2024-03-25T18:26:45.822 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:26:45.822676+0000 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:26:45.822676+0000 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:45.823 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.821+0000 7ff73d498640 1 Processor -- start 2024-03-25T18:26:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.821+0000 7ff73d498640 1 -- start start 2024-03-25T18:26:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.821+0000 7ff73d498640 10 monclient: init 2024-03-25T18:26:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 10 monclient(hunting): picked mon.noname-a con 0x7ff7380a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 10 monclient(hunting): picked mon.noname-c con 0x7ff73813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:45.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 10 monclient(hunting): picked mon.noname-b con 0x7ff7381430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:45.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff738142520 con 0x7ff7380a98f0 2024-03-25T18:26:45.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7380b28a0 con 0x7ff73813f710 2024-03-25T18:26:45.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7380b2a30 con 0x7ff7381430c0 2024-03-25T18:26:45.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:45.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.822+0000 7ff73d498640 10 monclient(hunting): authenticate will time out at 7437.111422s 2024-03-25T18:26:45.830 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.828+0000 7ff72ffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff73813f710 0x7ff7381429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:57940/0 (socket says 172.21.15.102:57940) 2024-03-25T18:26:45.830 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.828+0000 7ff72ffff640 1 -- 172.21.15.102:0/2143975485 learned_addr learned my addr 172.21.15.102:0/2143975485 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:45.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.839+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853830536 0 0) 0x7ff7380b28a0 con 0x7ff73813f710 2024-03-25T18:26:45.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.839+0000 7ff7367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:45.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.839+0000 7ff7367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:45.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.839+0000 7ff7367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:45.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.839+0000 7ff7367fc640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:26:45.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.839+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff724003520 con 0x7ff73813f710 2024-03-25T18:26:45.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1987913177 0 0) 0x7ff724003520 con 0x7ff73813f710 2024-03-25T18:26:45.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 >> v1:172.21.15.102:6790/0 conn(0x7ff7381430c0 legacy=0x7ff73814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 >> v1:172.21.15.102:6789/0 conn(0x7ff7380a98f0 legacy=0x7ff7380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:45.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 1 monclient: found mon.noname-c 2024-03-25T18:26:45.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:45.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:26:45.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7380b44c0 con 0x7ff73813f710 2024-03-25T18:26:45.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:45.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:45.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff7367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:45.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff73d498640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:26:45.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.852+0000 7ff73d498640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.854+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff720001690 con 0x7ff73813f710 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.854+0000 7ff7367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.854+0000 7ff7367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.854+0000 7ff7367fc640 10 monclient: dump: 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:45.856 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:45.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.854+0000 7ff73d498640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:45.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.877+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff720001920 con 0x7ff73813f710 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.877+0000 7ff7367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.877+0000 7ff7367fc640 1 -- 172.21.15.102:0/2143975485 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff720001c30 con 0x7ff73813f710 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.877+0000 7ff73d498640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.877+0000 7ff73d498640 4 set_mon_vals no callback set 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.877+0000 7ff7367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.877+0000 7ff7367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.877+0000 7ff7367fc640 10 monclient: dump: 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:45.879 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:45.880 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:45.880 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:45.880 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:45.880 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 10 monclient: shutdown 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 1 -- 172.21.15.102:0/2143975485 >> v1:172.21.15.146:6789/0 conn(0x7ff73813f710 legacy=0x7ff7381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 1 -- 172.21.15.102:0/2143975485 shutdown_connections 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 1 -- 172.21.15.102:0/2143975485 >> 172.21.15.102:0/2143975485 conn(0x7ff7380b7ea0 msgr2=0x7ff7380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 1 -- 172.21.15.102:0/2143975485 shutdown_connections 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 1 -- 172.21.15.102:0/2143975485 wait complete. 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 10 monclient: build_initial_monmap 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.878+0000 7ff73d498640 10 monclient: monmap: 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:45.883 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 librados: starting msgr at 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 librados: starting objecter 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 Processor -- start 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 -- start start 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 librados: setting wanted keys 2024-03-25T18:26:45.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 librados: calling monclient init 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 10 monclient: init 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 10 monclient(hunting): picked mon.noname-a con 0x7ff7380a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 10 monclient(hunting): picked mon.noname-b con 0x7ff73813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 10 monclient(hunting): picked mon.noname-c con 0x7ff7381430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7381333c0 con 0x7ff7380a98f0 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff738135100 con 0x7ff73813f710 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff738132f50 con 0x7ff7381430c0 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:45.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.879+0000 7ff73d498640 10 monclient(hunting): authenticate will time out at 7437.168812s 2024-03-25T18:26:45.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.886+0000 7ff736ffd640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff7380a98f0 0x7ff7381daa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54328/0 (socket says 172.21.15.102:54328) 2024-03-25T18:26:45.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.886+0000 7ff736ffd640 1 -- 172.21.15.102:0/3410302176 learned_addr learned my addr 172.21.15.102:0/3410302176 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:45.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.887+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7380a98f0 2024-03-25T18:26:45.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.887+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:45.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.891+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff73813f710 2024-03-25T18:26:45.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.891+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:45.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.895+0000 7ff72f7fe640 10 client.?.objecter ms_handle_connect 0x7ff7381430c0 2024-03-25T18:26:45.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.895+0000 7ff72f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:45.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163730053 0 0) 0x7ff738135100 con 0x7ff73813f710 2024-03-25T18:26:45.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:45.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:45.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:45.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:26:45.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff718004eb0 con 0x7ff73813f710 2024-03-25T18:26:45.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585697833 0 0) 0x7ff7381333c0 con 0x7ff7380a98f0 2024-03-25T18:26:45.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:45.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:45.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:45.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:26:45.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.902+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff738135100 con 0x7ff7380a98f0 2024-03-25T18:26:45.912 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.910+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927576417 0 0) 0x7ff738132f50 con 0x7ff7381430c0 2024-03-25T18:26:45.912 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.910+0000 7ff72f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:45.912 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.910+0000 7ff72f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:45.912 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.910+0000 7ff72f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:45.912 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.910+0000 7ff72f7fe640 10 monclient(hunting): my global_id is 4250 2024-03-25T18:26:45.912 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.910+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7381333c0 con 0x7ff7381430c0 2024-03-25T18:26:45.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.917+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2854448017 0 0) 0x7ff718004eb0 con 0x7ff73813f710 2024-03-25T18:26:45.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.918+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff738132f50 con 0x7ff73813f710 2024-03-25T18:26:45.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.918+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff720001650 con 0x7ff73813f710 2024-03-25T18:26:45.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.918+0000 7ff72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:45.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.918+0000 7ff72f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:45.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.918+0000 7ff72f7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:45.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.918+0000 7ff72f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.922+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1365340200 0 0) 0x7ff738135100 con 0x7ff7380a98f0 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.922+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff718004eb0 con 0x7ff7380a98f0 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.922+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff728001970 con 0x7ff7380a98f0 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.922+0000 7ff72f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.922+0000 7ff72f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.922+0000 7ff72f7fe640 10 monclient(hunting): dump: 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:45.924 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:45.925 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:45.925 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:45.925 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:45.925 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:45.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2047494750 0 0) 0x7ff738132f50 con 0x7ff73813f710 2024-03-25T18:26:45.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 >> v1:172.21.15.102:6790/0 conn(0x7ff7381430c0 legacy=0x7ff7381e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 >> v1:172.21.15.102:6789/0 conn(0x7ff7380a98f0 legacy=0x7ff7381daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:45.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:45.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 1 monclient: found mon.b 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7381330e0 con 0x7ff73813f710 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff72f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 10 monclient: _renew_subs 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff738135420 con 0x7ff73813f710 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 10 monclient: _renew_subs 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:45.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73813b930 con 0x7ff73813f710 2024-03-25T18:26:45.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.927+0000 7ff73d498640 1 librados: init done 2024-03-25T18:26:45.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.928+0000 7ff73d498640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:45.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.928+0000 7ff73d498640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:45.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.928+0000 7ff73d498640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:45.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:45.928+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7380627b0 con 0x7ff73813f710 2024-03-25T18:26:46.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.220+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7200010e0 con 0x7ff73813f710 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.220+0000 7ff72f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.220+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff720004d90 con 0x7ff73813f710 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.220+0000 7ff72f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.220+0000 7ff72f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.220+0000 7ff735ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.220+0000 7ff72f7fe640 10 monclient: dump: 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:46.222 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.223 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:46.223 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:46.223 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:46.223 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:46.223 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:46.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.221+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7ff72001ad60 con 0x7ff73813f710 2024-03-25T18:26:46.224 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.221+0000 7ff736ffd640 10 monclient: get_auth_request con 0x7ff71806df20 auth_method 0 2024-03-25T18:26:46.224 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.221+0000 7ff72f7fe640 10 client.4264.objecter ms_handle_connect 0x7ff71806df20 2024-03-25T18:26:46.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.227+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7ff7200548b0 con 0x7ff73813f710 2024-03-25T18:26:46.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.227+0000 7ff72f7fe640 10 client.4264.objecter ms_dispatch 0x7ff738000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:46.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.227+0000 7ff72f7fe640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:46.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.227+0000 7ff72f7fe640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:46.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.227+0000 7ff72f7fe640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T18:26:46.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.228+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7200503c0 con 0x7ff73813f710 2024-03-25T18:26:46.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.228+0000 7ff72f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:46.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.228+0000 7ff72f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:46.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.446+0000 7ff73d498640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:46.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.446+0000 7ff73d498640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:46.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.446+0000 7ff73d498640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:26:46.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.446+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff73814f300 con 0x7ff73813f710 2024-03-25T18:26:46.472 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.471+0000 7ff72f7fe640 1 -- 172.21.15.102:0/3410302176 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7ff7200522f0 con 0x7ff73813f710 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.471+0000 7ff72f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.471+0000 7ff72f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:26:46.473 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.473+0000 7ff73d498640 10 librados: watch_flush enter 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.473+0000 7ff73d498640 10 librados: watch_flush exit 2024-03-25T18:26:46.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.473+0000 7ff73d498640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.473+0000 7ff73d498640 10 client.4264.objecter successfully canceled tick 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.473+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 >> v1:172.21.15.146:6800/3848252714 conn(0x7ff71806df20 legacy=0x7ff7180703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.473+0000 7ff73d498640 10 monclient: shutdown 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.473+0000 7ff73d498640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.473+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 >> v1:172.21.15.146:6789/0 conn(0x7ff73813f710 legacy=0x7ff7381db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.474+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 shutdown_connections 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.474+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 >> 172.21.15.102:0/3410302176 conn(0x7ff7380b7ea0 msgr2=0x7ff738074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.474+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 shutdown_connections 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.474+0000 7ff73d498640 1 -- 172.21.15.102:0/3410302176 wait complete. 2024-03-25T18:26:46.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:46.474+0000 7ff73d498640 1 librados: shutdown 2024-03-25T18:26:46.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:53.295 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:53.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.409+0000 7f10726c5640 10 monclient: get_monmap_and_config 2024-03-25T18:26:53.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.409+0000 7f10726c5640 10 monclient: build_initial_monmap 2024-03-25T18:26:53.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.409+0000 7f10726c5640 10 monclient: monmap: 2024-03-25T18:26:53.411 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:53.411 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:53.411 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:26:53.410783+0000 2024-03-25T18:26:53.411 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:26:53.410783+0000 2024-03-25T18:26:53.412 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:53.412 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:53.412 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:53.412 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:26:53.412 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:26:53.412 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.409+0000 7f10726c5640 1 Processor -- start 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 1 -- start start 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 10 monclient: init 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 10 monclient(hunting): picked mon.noname-c con 0x7f106c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 10 monclient(hunting): picked mon.noname-a con 0x7f106c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 10 monclient(hunting): picked mon.noname-b con 0x7f106c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c142520 con 0x7f106c13f710 2024-03-25T18:26:53.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c0b28a0 con 0x7f106c0a98f0 2024-03-25T18:26:53.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c0b2a30 con 0x7f106c1430c0 2024-03-25T18:26:53.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:53.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.410+0000 7f10726c5640 10 monclient(hunting): authenticate will time out at 7444.699791s 2024-03-25T18:26:53.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.422+0000 7f106b7fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f106c13f710 0x7f106c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43472/0 (socket says 172.21.15.102:43472) 2024-03-25T18:26:53.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.422+0000 7f106b7fe640 1 -- 172.21.15.102:0/2121643774 learned_addr learned my addr 172.21.15.102:0/2121643774 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:53.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.434+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096150207 0 0) 0x7f106c0b2a30 con 0x7f106c1430c0 2024-03-25T18:26:53.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.434+0000 7f106affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.434+0000 7f106affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.434+0000 7f106affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.434+0000 7f106affd640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:26:53.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.434+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1054003520 con 0x7f106c1430c0 2024-03-25T18:26:53.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.438+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752936380 0 0) 0x7f106c142520 con 0x7f106c13f710 2024-03-25T18:26:53.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.438+0000 7f106affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.438+0000 7f106affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.438+0000 7f106affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.438+0000 7f106affd640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:26:53.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.438+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106c0b2a30 con 0x7f106c13f710 2024-03-25T18:26:53.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2719615024 0 0) 0x7f1054003520 con 0x7f106c1430c0 2024-03-25T18:26:53.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 >> v1:172.21.15.146:6789/0 conn(0x7f106c0a98f0 legacy=0x7f106c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 >> v1:172.21.15.102:6789/0 conn(0x7f106c13f710 legacy=0x7f106c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 1 monclient: found mon.noname-b 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c0b44c0 con 0x7f106c1430c0 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 10 monclient: _finish_auth 0 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1064001a10 con 0x7f106c1430c0 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f106affd640 10 monclient: dump: 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:53.443 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f10726c5640 5 monclient: authenticate success, global_id 4253 2024-03-25T18:26:53.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.441+0000 7f10726c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.451+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1064001bf0 con 0x7f106c1430c0 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.451+0000 7f106affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.451+0000 7f106affd640 1 -- 172.21.15.102:0/2121643774 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1064002000 con 0x7f106c1430c0 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.451+0000 7f10726c5640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.451+0000 7f10726c5640 4 set_mon_vals no callback set 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.451+0000 7f106affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.451+0000 7f106affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.451+0000 7f106affd640 10 monclient: dump: 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:53.453 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.454 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.454 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.454 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:53.454 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:53.454 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:53.454 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:53.454 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:53.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.452+0000 7f10726c5640 10 monclient: shutdown 2024-03-25T18:26:53.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.452+0000 7f10726c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.452+0000 7f10726c5640 1 -- 172.21.15.102:0/2121643774 >> v1:172.21.15.102:6790/0 conn(0x7f106c1430c0 legacy=0x7f106c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.452+0000 7f10726c5640 1 -- 172.21.15.102:0/2121643774 shutdown_connections 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.452+0000 7f10726c5640 1 -- 172.21.15.102:0/2121643774 >> 172.21.15.102:0/2121643774 conn(0x7f106c0b7ea0 msgr2=0x7f106c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.452+0000 7f10726c5640 1 -- 172.21.15.102:0/2121643774 shutdown_connections 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.452+0000 7f10726c5640 1 -- 172.21.15.102:0/2121643774 wait complete. 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 10 monclient: build_initial_monmap 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 10 monclient: monmap: 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:26:53.457 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 1 librados: starting msgr at 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 1 librados: starting objecter 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 1 Processor -- start 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 1 -- start start 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 1 librados: setting wanted keys 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 1 librados: calling monclient init 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.453+0000 7f10726c5640 10 monclient: init 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 10 monclient(hunting): picked mon.noname-b con 0x7f106c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 10 monclient(hunting): picked mon.noname-c con 0x7f106c13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:26:53.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 10 monclient(hunting): picked mon.noname-a con 0x7f106c1430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:26:53.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c138b40 con 0x7f106c1430c0 2024-03-25T18:26:53.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c138e20 con 0x7f106c0a98f0 2024-03-25T18:26:53.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c139100 con 0x7f106c13f710 2024-03-25T18:26:53.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:53.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.454+0000 7f10726c5640 10 monclient(hunting): authenticate will time out at 7444.743526s 2024-03-25T18:26:53.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.467+0000 7f1070c62640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f106c1430c0 0x7f106c1e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43480/0 (socket says 172.21.15.102:43480) 2024-03-25T18:26:53.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.467+0000 7f1070c62640 1 -- 172.21.15.102:0/2463419307 learned_addr learned my addr 172.21.15.102:0/2463419307 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:26:53.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.469+0000 7f1068ff9640 10 client.?.objecter ms_handle_connect 0x7f106c1430c0 2024-03-25T18:26:53.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.469+0000 7f1068ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:53.480 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.479+0000 7f1068ff9640 10 client.?.objecter ms_handle_connect 0x7f106c0a98f0 2024-03-25T18:26:53.480 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.479+0000 7f1068ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:53.490 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.489+0000 7f1068ff9640 10 client.?.objecter ms_handle_connect 0x7f106c13f710 2024-03-25T18:26:53.490 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.489+0000 7f1068ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:53.491 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.489+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136223002 0 0) 0x7f106c138b40 con 0x7f106c1430c0 2024-03-25T18:26:53.491 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.489+0000 7f1068ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.491 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.489+0000 7f1068ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.491 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.489+0000 7f1068ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.491 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.489+0000 7f1068ff9640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:26:53.491 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.489+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1058004eb0 con 0x7f106c1430c0 2024-03-25T18:26:53.493 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.492+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163760381 0 0) 0x7f106c138e20 con 0x7f106c0a98f0 2024-03-25T18:26:53.494 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.492+0000 7f1068ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.494 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.492+0000 7f1068ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.494 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.492+0000 7f1068ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.494 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.492+0000 7f1068ff9640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:26:53.494 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.492+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106c138b40 con 0x7f106c0a98f0 2024-03-25T18:26:53.507 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.505+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009865942 0 0) 0x7f106c139100 con 0x7f106c13f710 2024-03-25T18:26:53.507 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.505+0000 7f1068ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:53.507 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.505+0000 7f1068ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:53.507 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.505+0000 7f1068ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:53.507 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.505+0000 7f1068ff9640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:26:53.507 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.505+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106c138e20 con 0x7f106c13f710 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.507+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338901828 0 0) 0x7f106c138b40 con 0x7f106c0a98f0 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.507+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f106c139100 con 0x7f106c0a98f0 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.507+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f105c0018a0 con 0x7f106c0a98f0 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.507+0000 7f1068ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.507+0000 7f1068ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.507+0000 7f1068ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:53.509 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:53.510 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:53.510 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:53.510 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:53.510 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.507+0000 7f1068ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:53.515 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.513+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153881179 0 0) 0x7f106c138e20 con 0x7f106c13f710 2024-03-25T18:26:53.515 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.513+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f106c138b40 con 0x7f106c13f710 2024-03-25T18:26:53.515 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.513+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1060001930 con 0x7f106c13f710 2024-03-25T18:26:53.515 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.513+0000 7f1068ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.515 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.513+0000 7f1068ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:53.515 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.513+0000 7f1068ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:53.515 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:53.516 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:53.516 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.516 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.516 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.516 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:53.517 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:53.517 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:53.517 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:53.517 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.514+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 405570764 0 0) 0x7f1058004eb0 con 0x7f106c1430c0 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.514+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f106c138e20 con 0x7f106c1430c0 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.514+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1064001fc0 con 0x7f106c1430c0 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.514+0000 7f1068ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.514+0000 7f1068ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.514+0000 7f1068ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.518 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.519 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.519 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:53.519 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:53.519 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:53.519 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:53.519 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:53.526 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2609757726 0 0) 0x7f106c138b40 con 0x7f106c13f710 2024-03-25T18:26:53.526 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 >> v1:172.21.15.146:6789/0 conn(0x7f106c0a98f0 legacy=0x7f106c137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.526 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 >> v1:172.21.15.102:6789/0 conn(0x7f106c1430c0 legacy=0x7f106c1e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:53.526 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:53.526 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 1 monclient: found mon.c 2024-03-25T18:26:53.526 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:53.526 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:53.526 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c1397a0 con 0x7f106c13f710 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f1068ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 5 monclient: authenticate success, global_id 4256 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 10 monclient: _renew_subs 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c136aa0 con 0x7f106c13f710 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 10 monclient: _renew_subs 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c13c320 con 0x7f106c13f710 2024-03-25T18:26:53.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.524+0000 7f10726c5640 1 librados: init done 2024-03-25T18:26:53.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.525+0000 7f10726c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:53.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.525+0000 7f10726c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:53.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.525+0000 7f10726c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:53.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.525+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106c1395b0 con 0x7f106c13f710 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.824+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10600047e0 con 0x7f106c13f710 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.824+0000 7f1068ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.824+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1060001d80 con 0x7f106c13f710 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.824+0000 7f1068ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.824+0000 7f1068ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.824+0000 7f1068ff9640 10 monclient: dump: 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:26:53.826 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.827 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:26:53.827 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:53.827 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:26:53.827 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:26:53.827 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:26:53.827 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:26:53.827 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:26:53.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.824+0000 7f106a7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:53.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.825+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f106001a9a0 con 0x7f106c13f710 2024-03-25T18:26:53.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.825+0000 7f106bfff640 10 monclient: get_auth_request con 0x7f105806e160 auth_method 0 2024-03-25T18:26:53.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.826+0000 7f1068ff9640 10 client.4256.objecter ms_handle_connect 0x7f105806e160 2024-03-25T18:26:53.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.831+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f10600546b0 con 0x7f106c13f710 2024-03-25T18:26:53.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.831+0000 7f1068ff9640 10 client.4256.objecter ms_dispatch 0x7f106c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:53.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.831+0000 7f1068ff9640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:53.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.831+0000 7f1068ff9640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:53.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.831+0000 7f1068ff9640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-25T18:26:53.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.832+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f106004fff0 con 0x7f106c13f710 2024-03-25T18:26:53.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.832+0000 7f1068ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:53.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:53.832+0000 7f1068ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:54.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.078+0000 7f10726c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:54.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.078+0000 7f10726c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:54.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.078+0000 7f10726c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:26:54.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.078+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f106c139930 con 0x7f106c13f710 2024-03-25T18:26:54.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.137+0000 7f1068ff9640 1 -- 172.21.15.102:0/2463419307 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f10600520f0 con 0x7f106c13f710 2024-03-25T18:26:54.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.137+0000 7f1068ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:54.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.137+0000 7f1068ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:54.140 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:26:54.140 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:54.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 10 librados: watch_flush enter 2024-03-25T18:26:54.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 10 librados: watch_flush exit 2024-03-25T18:26:54.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-25T18:26:54.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 10 client.4256.objecter successfully canceled tick 2024-03-25T18:26:54.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 >> v1:172.21.15.146:6800/3848252714 conn(0x7f105806e160 legacy=0x7f1058070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 10 monclient: shutdown 2024-03-25T18:26:54.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:54.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 >> v1:172.21.15.102:6790/0 conn(0x7f106c13f710 legacy=0x7f106c137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 shutdown_connections 2024-03-25T18:26:54.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.140+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 >> 172.21.15.102:0/2463419307 conn(0x7f106c0b7ea0 msgr2=0x7f106c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:54.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.141+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 shutdown_connections 2024-03-25T18:26:54.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.141+0000 7f10726c5640 1 -- 172.21.15.102:0/2463419307 wait complete. 2024-03-25T18:26:54.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:26:54.141+0000 7f10726c5640 1 librados: shutdown 2024-03-25T18:26:54.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:57.981 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:14.216 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.051 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:14.287 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.079 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:14.293 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.137 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:14.350 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.146 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:14.382 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.208 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:14.444 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.310 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:14.546 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.372 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:14.585 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.472 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:14.708 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:00.960 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:01.074 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.074+0000 7f49c3a4a640 10 monclient: get_monmap_and_config 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.074+0000 7f49c3a4a640 10 monclient: build_initial_monmap 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.074+0000 7f49c3a4a640 10 monclient: monmap: 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:27:01.074856+0000 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:27:01.074856+0000 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:01.075 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:27:01.076 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:27:01.076 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:01.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.074+0000 7f49c3a4a640 1 Processor -- start 2024-03-25T18:27:01.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.075+0000 7f49c3a4a640 1 -- start start 2024-03-25T18:27:01.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.075+0000 7f49c3a4a640 10 monclient: init 2024-03-25T18:27:01.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:01.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:01.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f49bc0a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:01.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f49bc13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:01.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f49bc1430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:01.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc142520 con 0x7f49bc1430c0 2024-03-25T18:27:01.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc0b28a0 con 0x7f49bc13f710 2024-03-25T18:27:01.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc0b2a30 con 0x7f49bc0a98f0 2024-03-25T18:27:01.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:01.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.076+0000 7f49c3a4a640 10 monclient(hunting): authenticate will time out at 7452.364045s 2024-03-25T18:27:01.086 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.086+0000 7f49c17e6640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f49bc0a98f0 0x7f49bc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:40588/0 (socket says 172.21.15.102:40588) 2024-03-25T18:27:01.086 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.086+0000 7f49c17e6640 1 -- 172.21.15.102:0/3716764330 learned_addr learned my addr 172.21.15.102:0/3716764330 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:01.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.101+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900751375 0 0) 0x7f49bc0b2a30 con 0x7f49bc0a98f0 2024-03-25T18:27:01.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.101+0000 7f49b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.101+0000 7f49b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.101+0000 7f49b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.101+0000 7f49b3fff640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:27:01.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.101+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a8003520 con 0x7f49bc0a98f0 2024-03-25T18:27:01.103 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.103+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815637388 0 0) 0x7f49bc142520 con 0x7f49bc1430c0 2024-03-25T18:27:01.103 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.103+0000 7f49b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.103 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.103+0000 7f49b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.103 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.103+0000 7f49b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.103+0000 7f49b3fff640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:27:01.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.103+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc0b2a30 con 0x7f49bc1430c0 2024-03-25T18:27:01.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021542357 0 0) 0x7f49bc0b2a30 con 0x7f49bc1430c0 2024-03-25T18:27:01.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 >> v1:172.21.15.102:6790/0 conn(0x7f49bc0a98f0 legacy=0x7f49bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 >> v1:172.21.15.146:6789/0 conn(0x7f49bc13f710 legacy=0x7f49bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:01.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 1 monclient: found mon.noname-a 2024-03-25T18:27:01.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:01.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc0b44c0 con 0x7f49bc1430c0 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 10 monclient: _finish_auth 0 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f49b40019f0 con 0x7f49bc1430c0 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49b3fff640 10 monclient: dump: 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.111 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.112 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.112 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:01.112 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:01.112 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:01.112 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:01.112 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:01.112 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49c3a4a640 5 monclient: authenticate success, global_id 4275 2024-03-25T18:27:01.112 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.109+0000 7f49c3a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.131+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49b4001b50 con 0x7f49bc1430c0 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.131+0000 7f49b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.131+0000 7f49b3fff640 1 -- 172.21.15.102:0/3716764330 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f49b4001f20 con 0x7f49bc1430c0 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.131+0000 7f49c3a4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.131+0000 7f49c3a4a640 4 set_mon_vals no callback set 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.131+0000 7f49b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.131+0000 7f49b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.131+0000 7f49b3fff640 10 monclient: dump: 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.132 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.133 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:01.133 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:01.133 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:01.133 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:01.133 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:01.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.132+0000 7f49c3a4a640 10 monclient: shutdown 2024-03-25T18:27:01.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.132+0000 7f49c3a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:01.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.132+0000 7f49c3a4a640 1 -- 172.21.15.102:0/3716764330 >> v1:172.21.15.102:6789/0 conn(0x7f49bc1430c0 legacy=0x7f49bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.132+0000 7f49c3a4a640 1 -- 172.21.15.102:0/3716764330 shutdown_connections 2024-03-25T18:27:01.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.132+0000 7f49c3a4a640 1 -- 172.21.15.102:0/3716764330 >> 172.21.15.102:0/3716764330 conn(0x7f49bc0b7ea0 msgr2=0x7f49bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:01.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.132+0000 7f49c3a4a640 1 -- 172.21.15.102:0/3716764330 shutdown_connections 2024-03-25T18:27:01.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.132+0000 7f49c3a4a640 1 -- 172.21.15.102:0/3716764330 wait complete. 2024-03-25T18:27:01.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.133+0000 7f49c3a4a640 10 monclient: build_initial_monmap 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.133+0000 7f49c3a4a640 10 monclient: monmap: 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.133+0000 7f49c3a4a640 1 librados: starting msgr at 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.133+0000 7f49c3a4a640 1 librados: starting objecter 2024-03-25T18:27:01.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.133+0000 7f49c3a4a640 1 Processor -- start 2024-03-25T18:27:01.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 1 -- start start 2024-03-25T18:27:01.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 1 librados: setting wanted keys 2024-03-25T18:27:01.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 1 librados: calling monclient init 2024-03-25T18:27:01.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 10 monclient: init 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f49bc0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f49bc13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f49bc1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc139910 con 0x7f49bc0a98f0 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc0ab8b0 con 0x7f49bc13f710 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc1db470 con 0x7f49bc1430c0 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:01.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.134+0000 7f49c3a4a640 10 monclient(hunting): authenticate will time out at 7452.422775s 2024-03-25T18:27:01.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.145+0000 7f49c17e6640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f49bc0a98f0 0x7f49bc1daa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47676/0 (socket says 172.21.15.102:47676) 2024-03-25T18:27:01.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.145+0000 7f49c17e6640 1 -- 172.21.15.102:0/2250163914 learned_addr learned my addr 172.21.15.102:0/2250163914 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:01.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.147+0000 7f49b1ffb640 10 client.?.objecter ms_handle_connect 0x7f49bc1430c0 2024-03-25T18:27:01.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.147+0000 7f49b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:01.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.151+0000 7f49b1ffb640 10 client.?.objecter ms_handle_connect 0x7f49bc0a98f0 2024-03-25T18:27:01.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.151+0000 7f49b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:01.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.156+0000 7f49b1ffb640 10 client.?.objecter ms_handle_connect 0x7f49bc13f710 2024-03-25T18:27:01.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.156+0000 7f49b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:01.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.159+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672866947 0 0) 0x7f49bc139910 con 0x7f49bc0a98f0 2024-03-25T18:27:01.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.159+0000 7f49b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.159+0000 7f49b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.159+0000 7f49b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.159+0000 7f49b1ffb640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:27:01.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.160+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a0004eb0 con 0x7f49bc0a98f0 2024-03-25T18:27:01.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.163+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331843083 0 0) 0x7f49bc1db470 con 0x7f49bc1430c0 2024-03-25T18:27:01.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.163+0000 7f49b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.163+0000 7f49b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.163+0000 7f49b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.163+0000 7f49b1ffb640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:27:01.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.163+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc139910 con 0x7f49bc1430c0 2024-03-25T18:27:01.172 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.171+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402943315 0 0) 0x7f49bc0ab8b0 con 0x7f49bc13f710 2024-03-25T18:27:01.172 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.172+0000 7f49b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:01.172 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.172+0000 7f49b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:01.172 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.172+0000 7f49b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:01.172 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.172+0000 7f49b1ffb640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:27:01.172 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.172+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49bc1db470 con 0x7f49bc13f710 2024-03-25T18:27:01.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.175+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3723498299 0 0) 0x7f49a0004eb0 con 0x7f49bc0a98f0 2024-03-25T18:27:01.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.175+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49bc0ab8b0 con 0x7f49bc0a98f0 2024-03-25T18:27:01.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.175+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f49b8001990 con 0x7f49bc0a98f0 2024-03-25T18:27:01.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.175+0000 7f49b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.175+0000 7f49b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:01.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.175+0000 7f49b1ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:01.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.175+0000 7f49b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:01.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.189+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1459127691 0 0) 0x7f49bc139910 con 0x7f49bc1430c0 2024-03-25T18:27:01.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.189+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49a0004eb0 con 0x7f49bc1430c0 2024-03-25T18:27:01.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.189+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f49b4001e60 con 0x7f49bc1430c0 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.189+0000 7f49b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.189+0000 7f49b1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.189+0000 7f49b1ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:01.190 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:01.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.190+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2176037136 0 0) 0x7f49bc1db470 con 0x7f49bc13f710 2024-03-25T18:27:01.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.190+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49bc139910 con 0x7f49bc13f710 2024-03-25T18:27:01.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.190+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f49ac001860 con 0x7f49bc13f710 2024-03-25T18:27:01.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.190+0000 7f49b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.190+0000 7f49b1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:01.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.190+0000 7f49b1ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:01.192 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:01.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2919407071 0 0) 0x7f49bc0ab8b0 con 0x7f49bc0a98f0 2024-03-25T18:27:01.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 >> v1:172.21.15.102:6790/0 conn(0x7f49bc1430c0 legacy=0x7f49bc137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 >> v1:172.21.15.146:6789/0 conn(0x7f49bc13f710 legacy=0x7f49bc0b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:01.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 1 monclient: found mon.a 2024-03-25T18:27:01.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:01.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:01.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc1db110 con 0x7f49bc0a98f0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 5 monclient: authenticate success, global_id 4278 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 10 monclient: _renew_subs 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49bc0b0d50 con 0x7f49bc0a98f0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 10 monclient: _renew_subs 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49bc0b0750 con 0x7f49bc0a98f0 2024-03-25T18:27:01.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.195+0000 7f49c3a4a640 1 librados: init done 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.196+0000 7f49c3a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.196+0000 7f49c3a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.196+0000 7f49c3a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:01.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.196+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49bc1db600 con 0x7f49bc0a98f0 2024-03-25T18:27:01.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.499+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49b80047e0 con 0x7f49bc0a98f0 2024-03-25T18:27:01.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.499+0000 7f49b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:01.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.499+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f49b8004af0 con 0x7f49bc0a98f0 2024-03-25T18:27:01.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.499+0000 7f49b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:01.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.499+0000 7f49b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:01.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.499+0000 7f49b1ffb640 10 monclient: dump: 2024-03-25T18:27:01.501 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:01.501 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:01.501 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.502 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:01.502 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:01.502 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:01.502 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:01.502 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:01.502 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:01.503 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:01.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.499+0000 7f49b37fe640 4 set_mon_vals no callback set 2024-03-25T18:27:01.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.500+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f49b801ab80 con 0x7f49bc0a98f0 2024-03-25T18:27:01.504 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.500+0000 7f49c0fe5640 10 monclient: get_auth_request con 0x7f49a006e0f0 auth_method 0 2024-03-25T18:27:01.504 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.501+0000 7f49b1ffb640 10 client.4278.objecter ms_handle_connect 0x7f49a006e0f0 2024-03-25T18:27:01.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.506+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f49b80548c0 con 0x7f49bc0a98f0 2024-03-25T18:27:01.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.506+0000 7f49b1ffb640 10 client.4278.objecter ms_dispatch 0x7f49bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:01.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.506+0000 7f49b1ffb640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:01.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.506+0000 7f49b1ffb640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:01.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.506+0000 7f49b1ffb640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-25T18:27:01.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.506+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49b8050200 con 0x7f49bc0a98f0 2024-03-25T18:27:01.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.507+0000 7f49b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:01.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.507+0000 7f49b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:01.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.744+0000 7f49c3a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:01.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.744+0000 7f49c3a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:01.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.744+0000 7f49c3a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:01.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.744+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49bc1db2a0 con 0x7f49bc0a98f0 2024-03-25T18:27:01.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.765+0000 7f49b1ffb640 1 -- 172.21.15.102:0/2250163914 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f49b8052300 con 0x7f49bc0a98f0 2024-03-25T18:27:01.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.765+0000 7f49b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:01.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.765+0000 7f49b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:01.767 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:27:01.767 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:01.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 10 librados: watch_flush enter 2024-03-25T18:27:01.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 10 librados: watch_flush exit 2024-03-25T18:27:01.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-25T18:27:01.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 10 client.4278.objecter successfully canceled tick 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 >> v1:172.21.15.146:6800/3848252714 conn(0x7f49a006e0f0 legacy=0x7f49a00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 10 monclient: shutdown 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 >> v1:172.21.15.102:6789/0 conn(0x7f49bc0a98f0 legacy=0x7f49bc1daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 shutdown_connections 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 >> 172.21.15.102:0/2250163914 conn(0x7f49bc0b7ea0 msgr2=0x7f49bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 shutdown_connections 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 1 -- 172.21.15.102:0/2250163914 wait complete. 2024-03-25T18:27:01.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:01.768+0000 7f49c3a4a640 1 librados: shutdown 2024-03-25T18:27:01.781 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:08.589 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.703+0000 7fbc063a3640 10 monclient: get_monmap_and_config 2024-03-25T18:27:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.703+0000 7fbc063a3640 10 monclient: build_initial_monmap 2024-03-25T18:27:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.703+0000 7fbc063a3640 10 monclient: monmap: 2024-03-25T18:27:08.704 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:08.704 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:08.704 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:27:08.704466+0000 2024-03-25T18:27:08.704 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:27:08.704466+0000 2024-03-25T18:27:08.704 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:08.705 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:08.705 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:08.705 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:27:08.705 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:27:08.705 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:08.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.704+0000 7fbc063a3640 1 Processor -- start 2024-03-25T18:27:08.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.704+0000 7fbc063a3640 1 -- start start 2024-03-25T18:27:08.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.704+0000 7fbc063a3640 10 monclient: init 2024-03-25T18:27:08.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:08.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:08.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 10 monclient(hunting): picked mon.noname-b con 0x7fbc000a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:08.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0013f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:08.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 10 monclient(hunting): picked mon.noname-c con 0x7fbc001430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:08.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc00142520 con 0x7fbc0013f710 2024-03-25T18:27:08.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc000b28a0 con 0x7fbc001430c0 2024-03-25T18:27:08.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc000b2a30 con 0x7fbc000a98f0 2024-03-25T18:27:08.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:08.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.705+0000 7fbc063a3640 10 monclient(hunting): authenticate will time out at 7459.993716s 2024-03-25T18:27:08.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.715+0000 7fbc04940640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbc001430c0 0x7fbc0014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:56162/0 (socket says 172.21.15.102:56162) 2024-03-25T18:27:08.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.715+0000 7fbc04940640 1 -- 172.21.15.102:0/2648581564 learned_addr learned my addr 172.21.15.102:0/2648581564 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:08.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.736+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61398747 0 0) 0x7fbc00142520 con 0x7fbc0013f710 2024-03-25T18:27:08.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.736+0000 7fbbfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.736+0000 7fbbfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.736+0000 7fbbfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.736+0000 7fbbfeffd640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:27:08.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.736+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe8003520 con 0x7fbc0013f710 2024-03-25T18:27:08.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.738+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366820204 0 0) 0x7fbc000b28a0 con 0x7fbc001430c0 2024-03-25T18:27:08.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.738+0000 7fbbfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.738+0000 7fbbfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.738+0000 7fbbfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.738+0000 7fbbfeffd640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:27:08.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.738+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc00142520 con 0x7fbc001430c0 2024-03-25T18:27:08.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.741+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542380714 0 0) 0x7fbc000b2a30 con 0x7fbc000a98f0 2024-03-25T18:27:08.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.741+0000 7fbbfeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.741+0000 7fbbfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.741+0000 7fbbfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.741+0000 7fbbfeffd640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:27:08.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.741+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc000b28a0 con 0x7fbc000a98f0 2024-03-25T18:27:08.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 514132996 0 0) 0x7fbc000b28a0 con 0x7fbc000a98f0 2024-03-25T18:27:08.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 >> v1:172.21.15.146:6789/0 conn(0x7fbc001430c0 legacy=0x7fbc0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 >> v1:172.21.15.102:6789/0 conn(0x7fbc0013f710 legacy=0x7fbc001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:08.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 1 monclient: found mon.noname-b 2024-03-25T18:27:08.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:08.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc000b44c0 con 0x7fbc000a98f0 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbf00015b0 con 0x7fbc000a98f0 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbc063a3640 5 monclient: authenticate success, global_id 4265 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbc063a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:08.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.753+0000 7fbbfeffd640 10 monclient: dump: 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:08.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.754+0000 7fbc063a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:08.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf0001710 con 0x7fbc000a98f0 2024-03-25T18:27:08.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbbfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:08.761 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbbfeffd640 1 -- 172.21.15.102:0/2648581564 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbf0001b60 con 0x7fbc000a98f0 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbc063a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbc063a3640 4 set_mon_vals no callback set 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbbfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbbfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbbfeffd640 10 monclient: dump: 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:08.762 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:08.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbc063a3640 10 monclient: shutdown 2024-03-25T18:27:08.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbc063a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:08.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbc063a3640 1 -- 172.21.15.102:0/2648581564 >> v1:172.21.15.102:6790/0 conn(0x7fbc000a98f0 legacy=0x7fbc000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbc063a3640 1 -- 172.21.15.102:0/2648581564 shutdown_connections 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.761+0000 7fbc063a3640 1 -- 172.21.15.102:0/2648581564 >> 172.21.15.102:0/2648581564 conn(0x7fbc000b7ea0 msgr2=0x7fbc000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.762+0000 7fbc063a3640 1 -- 172.21.15.102:0/2648581564 shutdown_connections 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.762+0000 7fbc063a3640 1 -- 172.21.15.102:0/2648581564 wait complete. 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.762+0000 7fbc063a3640 10 monclient: build_initial_monmap 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.762+0000 7fbc063a3640 10 monclient: monmap: 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:27:08.765 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.763+0000 7fbc063a3640 1 librados: starting msgr at 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.763+0000 7fbc063a3640 1 librados: starting objecter 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.763+0000 7fbc063a3640 1 Processor -- start 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.763+0000 7fbc063a3640 1 -- start start 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.763+0000 7fbc063a3640 1 librados: setting wanted keys 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.763+0000 7fbc063a3640 1 librados: calling monclient init 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.763+0000 7fbc063a3640 10 monclient: init 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 10 monclient(hunting): picked mon.noname-c con 0x7fbc000a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0013f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 10 monclient(hunting): picked mon.noname-a con 0x7fbc001430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc001db6c0 con 0x7fbc001430c0 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc001daa90 con 0x7fbc0013f710 2024-03-25T18:27:08.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0014e8c0 con 0x7fbc000a98f0 2024-03-25T18:27:08.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:08.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.764+0000 7fbc063a3640 10 monclient(hunting): authenticate will time out at 7460.052317s 2024-03-25T18:27:08.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.780+0000 7fbbff7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbc0013f710 0x7fbc000b0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:56198/0 (socket says 172.21.15.102:56198) 2024-03-25T18:27:08.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.780+0000 7fbbff7fe640 1 -- 172.21.15.102:0/1582282635 learned_addr learned my addr 172.21.15.102:0/1582282635 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:08.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.781+0000 7fbbfcff9640 10 client.?.objecter ms_handle_connect 0x7fbc0013f710 2024-03-25T18:27:08.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.781+0000 7fbbfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:08.789 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.789+0000 7fbbfcff9640 10 client.?.objecter ms_handle_connect 0x7fbc000a98f0 2024-03-25T18:27:08.789 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.789+0000 7fbbfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:08.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.793+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789435838 0 0) 0x7fbc001daa90 con 0x7fbc0013f710 2024-03-25T18:27:08.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.793+0000 7fbbfcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.793+0000 7fbbfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.793+0000 7fbbfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.793+0000 7fbbfcff9640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:27:08.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.793+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbec004eb0 con 0x7fbc0013f710 2024-03-25T18:27:08.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.796+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457725902 0 0) 0x7fbc0014e8c0 con 0x7fbc000a98f0 2024-03-25T18:27:08.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.796+0000 7fbbfcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.796+0000 7fbbfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.796+0000 7fbbfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.796+0000 7fbbfcff9640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:27:08.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.796+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc001daa90 con 0x7fbc000a98f0 2024-03-25T18:27:08.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.798+0000 7fbbfcff9640 10 client.?.objecter ms_handle_connect 0x7fbc001430c0 2024-03-25T18:27:08.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.798+0000 7fbbfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:08.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.804+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070228416 0 0) 0x7fbc001db6c0 con 0x7fbc001430c0 2024-03-25T18:27:08.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.804+0000 7fbbfcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:08.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.804+0000 7fbbfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:08.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.804+0000 7fbbfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:08.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.804+0000 7fbbfcff9640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:27:08.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.804+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc0014e8c0 con 0x7fbc001430c0 2024-03-25T18:27:08.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.806+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1995439456 0 0) 0x7fbbec004eb0 con 0x7fbc0013f710 2024-03-25T18:27:08.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.806+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc001db6c0 con 0x7fbc0013f710 2024-03-25T18:27:08.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.806+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbf40019b0 con 0x7fbc0013f710 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.806+0000 7fbbfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.806+0000 7fbbfcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.806+0000 7fbbfcff9640 10 monclient(hunting): dump: 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:08.807 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:08.810 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.806+0000 7fbbfcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.810+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371419828 0 0) 0x7fbc001daa90 con 0x7fbc000a98f0 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.810+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbec004eb0 con 0x7fbc000a98f0 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.810+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbf0001a20 con 0x7fbc000a98f0 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.810+0000 7fbbfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:08.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.810+0000 7fbbfcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.810+0000 7fbbfcff9640 10 monclient(hunting): dump: 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:08.812 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2951858865 0 0) 0x7fbbec004eb0 con 0x7fbc000a98f0 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 >> v1:172.21.15.146:6789/0 conn(0x7fbc0013f710 legacy=0x7fbc000b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 >> v1:172.21.15.102:6789/0 conn(0x7fbc001430c0 legacy=0x7fbc001e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 1 monclient: found mon.c 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc000b14f0 con 0x7fbc000a98f0 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbbfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 5 monclient: authenticate success, global_id 4268 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 10 monclient: _renew_subs 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0014de20 con 0x7fbc000a98f0 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:08.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 10 monclient: _renew_subs 2024-03-25T18:27:08.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:08.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc001e32d0 con 0x7fbc000a98f0 2024-03-25T18:27:08.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.816+0000 7fbc063a3640 1 librados: init done 2024-03-25T18:27:08.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.817+0000 7fbc063a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:08.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.817+0000 7fbc063a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:08.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.817+0000 7fbc063a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:08.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:08.817+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc001da880 con 0x7fbc000a98f0 2024-03-25T18:27:09.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.143+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf00038a0 con 0x7fbc000a98f0 2024-03-25T18:27:09.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.143+0000 7fbbfcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:09.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.143+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbf0001c20 con 0x7fbc000a98f0 2024-03-25T18:27:09.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.143+0000 7fbbfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.143+0000 7fbbfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.143+0000 7fbbfcff9640 10 monclient: dump: 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:09.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.143+0000 7fbbfe7fc640 4 set_mon_vals no callback set 2024-03-25T18:27:09.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.144+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fbbf0019a20 con 0x7fbc000a98f0 2024-03-25T18:27:09.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.144+0000 7fbbff7fe640 10 monclient: get_auth_request con 0x7fbbec06dfb0 auth_method 0 2024-03-25T18:27:09.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.145+0000 7fbbfcff9640 10 client.4268.objecter ms_handle_connect 0x7fbbec06dfb0 2024-03-25T18:27:09.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.152+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fbbf0053660 con 0x7fbc000a98f0 2024-03-25T18:27:09.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.152+0000 7fbbfcff9640 10 client.4268.objecter ms_dispatch 0x7fbc00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:09.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.152+0000 7fbbfcff9640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:09.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.152+0000 7fbbfcff9640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:09.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.152+0000 7fbbfcff9640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-25T18:27:09.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.152+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbf004f170 con 0x7fbc000a98f0 2024-03-25T18:27:09.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.152+0000 7fbbfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:09.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.152+0000 7fbbfcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:09.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.386+0000 7fbc063a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:09.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.386+0000 7fbc063a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:09.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.386+0000 7fbc063a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:09.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.386+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc000627b0 con 0x7fbc000a98f0 2024-03-25T18:27:09.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.451+0000 7fbbfcff9640 1 -- 172.21.15.102:0/1582282635 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fbbf00510a0 con 0x7fbc000a98f0 2024-03-25T18:27:09.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.451+0000 7fbbfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:09.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.451+0000 7fbbfcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:09.453 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:27:09.453 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:09.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 10 librados: watch_flush enter 2024-03-25T18:27:09.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 10 librados: watch_flush exit 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 10 client.4268.objecter successfully canceled tick 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 >> v1:172.21.15.146:6800/3848252714 conn(0x7fbbec06dfb0 legacy=0x7fbbec070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 10 monclient: shutdown 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 >> v1:172.21.15.102:6790/0 conn(0x7fbc000a98f0 legacy=0x7fbc001dafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 shutdown_connections 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 >> 172.21.15.102:0/1582282635 conn(0x7fbc000b7ea0 msgr2=0x7fbc00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 shutdown_connections 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 1 -- 172.21.15.102:0/1582282635 wait complete. 2024-03-25T18:27:09.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:09.454+0000 7fbc063a3640 1 librados: shutdown 2024-03-25T18:27:09.467 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:16.275 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:16.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.371+0000 7f42b97d0640 10 monclient: get_monmap_and_config 2024-03-25T18:27:16.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.371+0000 7f42b97d0640 10 monclient: build_initial_monmap 2024-03-25T18:27:16.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.371+0000 7f42b97d0640 10 monclient: monmap: 2024-03-25T18:27:16.372 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:27:16.372226+0000 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:27:16.372226+0000 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 1 Processor -- start 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 1 -- start start 2024-03-25T18:27:16.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 10 monclient: init 2024-03-25T18:27:16.374 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 10 monclient(hunting): picked mon.noname-a con 0x7f42b40a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 10 monclient(hunting): picked mon.noname-c con 0x7f42b413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 10 monclient(hunting): picked mon.noname-b con 0x7f42b41430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.372+0000 7f42b97d0640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b4142520 con 0x7f42b40a98f0 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.373+0000 7f42b97d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b40b28a0 con 0x7f42b413f710 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.373+0000 7f42b97d0640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b40b2a30 con 0x7f42b41430c0 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.373+0000 7f42b97d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:16.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.373+0000 7f42b97d0640 10 monclient(hunting): authenticate will time out at 7467.660956s 2024-03-25T18:27:16.388 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.387+0000 7f42b37fe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f42b41430c0 0x7f42b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:46566/0 (socket says 172.21.15.102:46566) 2024-03-25T18:27:16.388 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.387+0000 7f42b37fe640 1 -- 172.21.15.102:0/734113934 learned_addr learned my addr 172.21.15.102:0/734113934 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727204147 0 0) 0x7f42b40b28a0 con 0x7f42b413f710 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f429c003520 con 0x7f42b413f710 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256873662 0 0) 0x7f42b40b2a30 con 0x7f42b41430c0 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.417 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:27:16.417 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.415+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b40b28a0 con 0x7f42b41430c0 2024-03-25T18:27:16.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.430+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579322795 0 0) 0x7f42b4142520 con 0x7f42b40a98f0 2024-03-25T18:27:16.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.430+0000 7f42b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.430+0000 7f42b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.430+0000 7f42b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.430+0000 7f42b1ffb640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:27:16.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.430+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b40b2a30 con 0x7f42b40a98f0 2024-03-25T18:27:16.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 783748200 0 0) 0x7f42b40b28a0 con 0x7f42b41430c0 2024-03-25T18:27:16.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 >> v1:172.21.15.146:6789/0 conn(0x7f42b413f710 legacy=0x7f42b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 >> v1:172.21.15.102:6789/0 conn(0x7f42b40a98f0 legacy=0x7f42b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:16.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 1 monclient: found mon.noname-b 2024-03-25T18:27:16.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:16.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:27:16.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b40b44c0 con 0x7f42b41430c0 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f42ac0019f0 con 0x7f42b41430c0 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b1ffb640 10 monclient: dump: 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.435 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.436 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.436 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:16.436 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:16.436 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:16.436 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:16.436 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:16.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b97d0640 5 monclient: authenticate success, global_id 4271 2024-03-25T18:27:16.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.433+0000 7f42b97d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:16.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42ac001b50 con 0x7f42b41430c0 2024-03-25T18:27:16.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:16.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b1ffb640 1 -- 172.21.15.102:0/734113934 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f42ac001f60 con 0x7f42b41430c0 2024-03-25T18:27:16.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b97d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:16.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b97d0640 4 set_mon_vals no callback set 2024-03-25T18:27:16.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:16.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b1ffb640 10 monclient: dump: 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:16.448 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:16.449 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b97d0640 10 monclient: shutdown 2024-03-25T18:27:16.449 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b97d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:16.449 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.447+0000 7f42b97d0640 1 -- 172.21.15.102:0/734113934 >> v1:172.21.15.102:6790/0 conn(0x7f42b41430c0 legacy=0x7f42b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.449 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.448+0000 7f42b97d0640 1 -- 172.21.15.102:0/734113934 shutdown_connections 2024-03-25T18:27:16.449 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.448+0000 7f42b97d0640 1 -- 172.21.15.102:0/734113934 >> 172.21.15.102:0/734113934 conn(0x7f42b40b7ea0 msgr2=0x7f42b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:16.449 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.448+0000 7f42b97d0640 1 -- 172.21.15.102:0/734113934 shutdown_connections 2024-03-25T18:27:16.449 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.448+0000 7f42b97d0640 1 -- 172.21.15.102:0/734113934 wait complete. 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.448+0000 7f42b97d0640 10 monclient: build_initial_monmap 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.448+0000 7f42b97d0640 10 monclient: monmap: 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:16.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.449+0000 7f42b97d0640 1 librados: starting msgr at 2024-03-25T18:27:16.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.449+0000 7f42b97d0640 1 librados: starting objecter 2024-03-25T18:27:16.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.449+0000 7f42b97d0640 1 Processor -- start 2024-03-25T18:27:16.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.449+0000 7f42b97d0640 1 -- start start 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.449+0000 7f42b97d0640 1 librados: setting wanted keys 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.449+0000 7f42b97d0640 1 librados: calling monclient init 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.449+0000 7f42b97d0640 10 monclient: init 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 10 monclient(hunting): picked mon.noname-c con 0x7f42b40a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 10 monclient(hunting): picked mon.noname-a con 0x7f42b413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 10 monclient(hunting): picked mon.noname-b con 0x7f42b41430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b41397d0 con 0x7f42b413f710 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b413af80 con 0x7f42b41430c0 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b41dba00 con 0x7f42b40a98f0 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:16.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.450+0000 7f42b97d0640 10 monclient(hunting): authenticate will time out at 7467.738535s 2024-03-25T18:27:16.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.456+0000 7f42b37fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f42b41430c0 0x7f42b41db2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:34728/0 (socket says 172.21.15.102:34728) 2024-03-25T18:27:16.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.456+0000 7f42b37fe640 1 -- 172.21.15.102:0/4078336325 learned_addr learned my addr 172.21.15.102:0/4078336325 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:16.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.457+0000 7f429b7fe640 10 client.?.objecter ms_handle_connect 0x7f42b41430c0 2024-03-25T18:27:16.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.457+0000 7f429b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:16.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.464+0000 7f429b7fe640 10 client.?.objecter ms_handle_connect 0x7f42b40a98f0 2024-03-25T18:27:16.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.464+0000 7f429b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:16.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.466+0000 7f429b7fe640 10 client.?.objecter ms_handle_connect 0x7f42b413f710 2024-03-25T18:27:16.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.466+0000 7f429b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:16.471 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.471+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936706981 0 0) 0x7f42b41397d0 con 0x7f42b413f710 2024-03-25T18:27:16.471 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.471+0000 7f429b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.472 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.471+0000 7f429b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.472 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.471+0000 7f429b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.472 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.471+0000 7f429b7fe640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:27:16.472 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.471+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a0004eb0 con 0x7f42b413f710 2024-03-25T18:27:16.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.480+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678481893 0 0) 0x7f42b413af80 con 0x7f42b41430c0 2024-03-25T18:27:16.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.480+0000 7f429b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.480+0000 7f429b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.480+0000 7f429b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.480+0000 7f429b7fe640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:27:16.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.480+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b41397d0 con 0x7f42b41430c0 2024-03-25T18:27:16.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.482+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314511990 0 0) 0x7f42b41dba00 con 0x7f42b40a98f0 2024-03-25T18:27:16.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.482+0000 7f429b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:16.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.482+0000 7f429b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:16.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.482+0000 7f429b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:16.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.482+0000 7f429b7fe640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:27:16.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.482+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b413af80 con 0x7f42b40a98f0 2024-03-25T18:27:16.491 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.491+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63476072 0 0) 0x7f42a0004eb0 con 0x7f42b413f710 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.491+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42b41dba00 con 0x7f42b413f710 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.491+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f42a8001950 con 0x7f42b413f710 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.491+0000 7f429b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.491+0000 7f429b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.491+0000 7f429b7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:16.492 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:16.493 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:16.493 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:16.493 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.491+0000 7f429b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:16.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.499+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1718758455 0 0) 0x7f42b41397d0 con 0x7f42b41430c0 2024-03-25T18:27:16.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.499+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42a0004eb0 con 0x7f42b41430c0 2024-03-25T18:27:16.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.499+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f42ac001ee0 con 0x7f42b41430c0 2024-03-25T18:27:16.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.499+0000 7f429b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.499+0000 7f429b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:16.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.499+0000 7f429b7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:16.501 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:16.505 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.505+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2743153854 0 0) 0x7f42b413af80 con 0x7f42b40a98f0 2024-03-25T18:27:16.505 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.505+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42b41397d0 con 0x7f42b40a98f0 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.505+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f42a4001830 con 0x7f42b40a98f0 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.505+0000 7f429b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.505+0000 7f429b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.505+0000 7f429b7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:16.506 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:16.507 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1729174402 0 0) 0x7f42a0004eb0 con 0x7f42b41430c0 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 >> v1:172.21.15.102:6790/0 conn(0x7f42b40a98f0 legacy=0x7f42b40b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 >> v1:172.21.15.102:6789/0 conn(0x7f42b413f710 legacy=0x7f42b41dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 1 monclient: found mon.b 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b413b260 con 0x7f42b41430c0 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:16.511 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f429b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 10 monclient: _renew_subs 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b40b74f0 con 0x7f42b41430c0 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 10 monclient: _renew_subs 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b412f780 con 0x7f42b41430c0 2024-03-25T18:27:16.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.510+0000 7f42b97d0640 1 librados: init done 2024-03-25T18:27:16.513 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.511+0000 7f42b97d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:16.513 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.511+0000 7f42b97d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:16.513 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.511+0000 7f42b97d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:16.513 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.511+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b40627b0 con 0x7f42b41430c0 2024-03-25T18:27:16.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.723+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42ac0020e0 con 0x7f42b41430c0 2024-03-25T18:27:16.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.723+0000 7f429b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:16.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.723+0000 7f42b17fa640 4 set_mon_vals no callback set 2024-03-25T18:27:16.790 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.790+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f42ac004fb0 con 0x7f42b41430c0 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.790+0000 7f429b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.790+0000 7f429b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.790+0000 7f429b7fe640 10 monclient: dump: 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:16.791 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:16.792 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:16.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.791+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f42ac01af80 con 0x7f42b41430c0 2024-03-25T18:27:16.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.791+0000 7f42b2ffd640 10 monclient: get_auth_request con 0x7f42a006e110 auth_method 0 2024-03-25T18:27:16.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.792+0000 7f429b7fe640 10 client.4288.objecter ms_handle_connect 0x7f42a006e110 2024-03-25T18:27:16.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.794+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f42ac054c00 con 0x7f42b41430c0 2024-03-25T18:27:16.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.794+0000 7f429b7fe640 10 client.4288.objecter ms_dispatch 0x7f42b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:16.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.794+0000 7f429b7fe640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:16.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.794+0000 7f429b7fe640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:16.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.794+0000 7f429b7fe640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T18:27:16.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.796+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42ac050540 con 0x7f42b41430c0 2024-03-25T18:27:16.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.796+0000 7f429b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:16.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:16.796+0000 7f429b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:17.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.040+0000 7f42b97d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:17.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.040+0000 7f42b97d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:17.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.040+0000 7f42b97d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:17.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.040+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42b414e8f0 con 0x7f42b41430c0 2024-03-25T18:27:17.079 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.078+0000 7f429b7fe640 1 -- 172.21.15.102:0/4078336325 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f42ac052640 con 0x7f42b41430c0 2024-03-25T18:27:17.079 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.078+0000 7f429b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:17.079 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.078+0000 7f429b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:17.079 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:27:17.079 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:17.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 10 librados: watch_flush enter 2024-03-25T18:27:17.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 10 librados: watch_flush exit 2024-03-25T18:27:17.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T18:27:17.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 10 client.4288.objecter successfully canceled tick 2024-03-25T18:27:17.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 >> v1:172.21.15.146:6800/3848252714 conn(0x7f42a006e110 legacy=0x7f42a00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 10 monclient: shutdown 2024-03-25T18:27:17.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:17.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 >> v1:172.21.15.146:6789/0 conn(0x7f42b41430c0 legacy=0x7f42b41db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 shutdown_connections 2024-03-25T18:27:17.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 >> 172.21.15.102:0/4078336325 conn(0x7f42b40b7ea0 msgr2=0x7f42b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:17.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 shutdown_connections 2024-03-25T18:27:17.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 1 -- 172.21.15.102:0/4078336325 wait complete. 2024-03-25T18:27:17.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:17.081+0000 7f42b97d0640 1 librados: shutdown 2024-03-25T18:27:17.093 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:17.946 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:34.181 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.994 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:34.230 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.050 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:34.284 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.053 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:34.266 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.065 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:34.278 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.078 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:34.314 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.085 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:34.321 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.117 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:34.330 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.147 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:34.383 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:23.901 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:24.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.017+0000 7f41ad606640 10 monclient: get_monmap_and_config 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.017+0000 7f41ad606640 10 monclient: build_initial_monmap 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.017+0000 7f41ad606640 10 monclient: monmap: 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:27:24.018010+0000 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:27:24.018010+0000 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:24.018 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:27:24.019 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:27:24.019 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 1 Processor -- start 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 1 -- start start 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 10 monclient: init 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 10 monclient(hunting): picked mon.noname-a con 0x7f41a80a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 10 monclient(hunting): picked mon.noname-b con 0x7f41a813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 10 monclient(hunting): picked mon.noname-c con 0x7f41a81430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.018+0000 7f41ad606640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a8142520 con 0x7f41a80a98f0 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.019+0000 7f41ad606640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80b28a0 con 0x7f41a81430c0 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.019+0000 7f41ad606640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a80b2a30 con 0x7f41a813f710 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.019+0000 7f41ad606640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:24.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.019+0000 7f41ad606640 10 monclient(hunting): authenticate will time out at 7475.306952s 2024-03-25T18:27:24.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.030+0000 7f41a67fc640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f41a813f710 0x7f41a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:37506/0 (socket says 172.21.15.102:37506) 2024-03-25T18:27:24.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.031+0000 7f41a67fc640 1 -- 172.21.15.102:0/838599046 learned_addr learned my addr 172.21.15.102:0/838599046 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:24.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.041+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807523574 0 0) 0x7f41a80b2a30 con 0x7f41a813f710 2024-03-25T18:27:24.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.041+0000 7f41a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.041+0000 7f41a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.041+0000 7f41a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.041+0000 7f41a5ffb640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:27:24.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.041+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4190003520 con 0x7f41a813f710 2024-03-25T18:27:24.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.046+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584774299 0 0) 0x7f41a8142520 con 0x7f41a80a98f0 2024-03-25T18:27:24.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.046+0000 7f41a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.046+0000 7f41a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.046+0000 7f41a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.046+0000 7f41a5ffb640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:27:24.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.046+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a80b2a30 con 0x7f41a80a98f0 2024-03-25T18:27:24.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.053+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032961741 0 0) 0x7f41a80b28a0 con 0x7f41a81430c0 2024-03-25T18:27:24.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.053+0000 7f41a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.053+0000 7f41a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.053+0000 7f41a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.053+0000 7f41a5ffb640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:27:24.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.053+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a8142520 con 0x7f41a81430c0 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801764942 0 0) 0x7f4190003520 con 0x7f41a813f710 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 >> v1:172.21.15.146:6789/0 conn(0x7f41a81430c0 legacy=0x7f41a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 >> v1:172.21.15.102:6789/0 conn(0x7f41a80a98f0 legacy=0x7f41a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 1 monclient: found mon.noname-b 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a80b44c0 con 0x7f41a813f710 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:24.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f419c001690 con 0x7f41a813f710 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41ad606640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41ad606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41a5ffb640 10 monclient: dump: 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:24.056 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:24.057 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:24.057 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:24.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.054+0000 7f41ad606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.060+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f419c0017f0 con 0x7f41a813f710 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.060+0000 7f41a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.060+0000 7f41a5ffb640 1 -- 172.21.15.102:0/838599046 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f419c001c00 con 0x7f41a813f710 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.060+0000 7f41ad606640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.060+0000 7f41ad606640 4 set_mon_vals no callback set 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.060+0000 7f41a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.060+0000 7f41a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.060+0000 7f41a5ffb640 10 monclient: dump: 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.061 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.062 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.062 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:24.062 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:24.062 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:24.062 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:24.062 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:24.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.061+0000 7f41ad606640 10 monclient: shutdown 2024-03-25T18:27:24.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.061+0000 7f41ad606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:24.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.061+0000 7f41ad606640 1 -- 172.21.15.102:0/838599046 >> v1:172.21.15.102:6790/0 conn(0x7f41a813f710 legacy=0x7f41a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.061+0000 7f41ad606640 1 -- 172.21.15.102:0/838599046 shutdown_connections 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.061+0000 7f41ad606640 1 -- 172.21.15.102:0/838599046 >> 172.21.15.102:0/838599046 conn(0x7f41a80b7ea0 msgr2=0x7f41a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.061+0000 7f41ad606640 1 -- 172.21.15.102:0/838599046 shutdown_connections 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.061+0000 7f41ad606640 1 -- 172.21.15.102:0/838599046 wait complete. 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.062+0000 7f41ad606640 10 monclient: build_initial_monmap 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.062+0000 7f41ad606640 10 monclient: monmap: 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:24.064 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 1 librados: starting msgr at 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 1 librados: starting objecter 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 1 Processor -- start 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 1 -- start start 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 1 librados: setting wanted keys 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 1 librados: calling monclient init 2024-03-25T18:27:24.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 10 monclient: init 2024-03-25T18:27:24.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:24.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:24.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 10 monclient(hunting): picked mon.noname-c con 0x7f41a80a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:24.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.063+0000 7f41ad606640 10 monclient(hunting): picked mon.noname-a con 0x7f41a813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:24.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.064+0000 7f41ad606640 10 monclient(hunting): picked mon.noname-b con 0x7f41a81430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:24.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.064+0000 7f41ad606640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a8137df0 con 0x7f41a813f710 2024-03-25T18:27:24.067 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.064+0000 7f41ad606640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a81337a0 con 0x7f41a81430c0 2024-03-25T18:27:24.067 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.064+0000 7f41ad606640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a8133930 con 0x7f41a80a98f0 2024-03-25T18:27:24.067 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.064+0000 7f41ad606640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:24.067 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.064+0000 7f41ad606640 10 monclient(hunting): authenticate will time out at 7475.352048s 2024-03-25T18:27:24.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.082+0000 7f41a77fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f41a81430c0 0x7f41a8133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:58630/0 (socket says 172.21.15.102:58630) 2024-03-25T18:27:24.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.082+0000 7f41a77fe640 1 -- 172.21.15.102:0/24046791 learned_addr learned my addr 172.21.15.102:0/24046791 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:24.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.083+0000 7f418f7fe640 10 client.?.objecter ms_handle_connect 0x7f41a81430c0 2024-03-25T18:27:24.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.083+0000 7f418f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:24.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.088+0000 7f418f7fe640 10 client.?.objecter ms_handle_connect 0x7f41a813f710 2024-03-25T18:27:24.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:24.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 10 client.?.objecter ms_handle_connect 0x7f41a80a98f0 2024-03-25T18:27:24.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:24.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297190610 0 0) 0x7f41a81337a0 con 0x7f41a81430c0 2024-03-25T18:27:24.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:27:24.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.089+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4194004eb0 con 0x7f41a81430c0 2024-03-25T18:27:24.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.097+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813345892 0 0) 0x7f41a8133930 con 0x7f41a80a98f0 2024-03-25T18:27:24.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.097+0000 7f418f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.097+0000 7f418f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.097+0000 7f418f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.097+0000 7f418f7fe640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:27:24.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.097+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a81337a0 con 0x7f41a80a98f0 2024-03-25T18:27:24.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.098+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671233196 0 0) 0x7f41a8137df0 con 0x7f41a813f710 2024-03-25T18:27:24.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.098+0000 7f418f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:24.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.098+0000 7f418f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:24.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.098+0000 7f418f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:24.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.098+0000 7f418f7fe640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:27:24.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.098+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a8133930 con 0x7f41a813f710 2024-03-25T18:27:24.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.107+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1437125898 0 0) 0x7f41a81337a0 con 0x7f41a80a98f0 2024-03-25T18:27:24.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.107+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41a8137df0 con 0x7f41a80a98f0 2024-03-25T18:27:24.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.107+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4198001970 con 0x7f41a80a98f0 2024-03-25T18:27:24.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.107+0000 7f418f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.107+0000 7f418f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.107+0000 7f418f7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:24.109 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.108+0000 7f418f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.115+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 277558003 0 0) 0x7f4194004eb0 con 0x7f41a81430c0 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.115+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41a81337a0 con 0x7f41a81430c0 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.115+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f41a0001c70 con 0x7f41a81430c0 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.115+0000 7f418f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.115+0000 7f418f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.115+0000 7f418f7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.116 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.117 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.117 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:24.117 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:24.117 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:24.117 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:24.117 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3218574861 0 0) 0x7f41a8137df0 con 0x7f41a80a98f0 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 >> v1:172.21.15.146:6789/0 conn(0x7f41a81430c0 legacy=0x7f41a8133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.118 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 >> v1:172.21.15.102:6789/0 conn(0x7f41a813f710 legacy=0x7f41a81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 1 monclient: found mon.c 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a81367d0 con 0x7f41a80a98f0 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f418f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f41ad606640 5 monclient: authenticate success, global_id 4280 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f41ad606640 10 monclient: _renew_subs 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f41ad606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.116+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a81369e0 con 0x7f41a80a98f0 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.117+0000 7f41ad606640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:24.119 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.117+0000 7f41ad606640 10 monclient: _renew_subs 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.117+0000 7f41ad606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.117+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a813b640 con 0x7f41a80a98f0 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.117+0000 7f41ad606640 1 librados: init done 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.118+0000 7f41ad606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.118+0000 7f41ad606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.118+0000 7f41ad606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:24.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.118+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a80627b0 con 0x7f41a80a98f0 2024-03-25T18:27:24.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.440+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4198003610 con 0x7f41a80a98f0 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.440+0000 7f418f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.440+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4198003920 con 0x7f41a80a98f0 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.440+0000 7f418f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.440+0000 7f418f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.440+0000 7f418f7fe640 10 monclient: dump: 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:24.441 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:24.442 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:24.442 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:24.442 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:24.442 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:24.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.440+0000 7f41a57fa640 4 set_mon_vals no callback set 2024-03-25T18:27:24.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.441+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f41980198f0 con 0x7f41a80a98f0 2024-03-25T18:27:24.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.441+0000 7f41a67fc640 10 monclient: get_auth_request con 0x7f419406df10 auth_method 0 2024-03-25T18:27:24.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.441+0000 7f418f7fe640 10 client.4280.objecter ms_handle_connect 0x7f419406df10 2024-03-25T18:27:24.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.446+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f41980535d0 con 0x7f41a80a98f0 2024-03-25T18:27:24.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.446+0000 7f418f7fe640 10 client.4280.objecter ms_dispatch 0x7f41a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:24.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.446+0000 7f418f7fe640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:24.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.446+0000 7f418f7fe640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:24.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.446+0000 7f418f7fe640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-25T18:27:24.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.446+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f419804ef10 con 0x7f41a80a98f0 2024-03-25T18:27:24.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.447+0000 7f418f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:24.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.447+0000 7f418f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:24.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.661+0000 7f41ad606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:24.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.661+0000 7f41ad606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:24.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.661+0000 7f41ad606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:24.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.661+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41a8074cc0 con 0x7f41a80a98f0 2024-03-25T18:27:24.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.699+0000 7f418f7fe640 1 -- 172.21.15.102:0/24046791 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f4198051010 con 0x7f41a80a98f0 2024-03-25T18:27:24.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.700+0000 7f418f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:24.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.700+0000 7f418f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:24.701 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:27:24.701 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.701+0000 7f41ad606640 10 librados: watch_flush enter 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.701+0000 7f41ad606640 10 librados: watch_flush exit 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.701+0000 7f41ad606640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.701+0000 7f41ad606640 10 client.4280.objecter successfully canceled tick 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 >> v1:172.21.15.146:6800/3848252714 conn(0x7f419406df10 legacy=0x7f41940703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 10 monclient: shutdown 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 >> v1:172.21.15.102:6790/0 conn(0x7f41a80a98f0 legacy=0x7f41a81392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 shutdown_connections 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 >> 172.21.15.102:0/24046791 conn(0x7f41a80b7ea0 msgr2=0x7f41a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 shutdown_connections 2024-03-25T18:27:24.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 1 -- 172.21.15.102:0/24046791 wait complete. 2024-03-25T18:27:24.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:24.702+0000 7f41ad606640 1 librados: shutdown 2024-03-25T18:27:24.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:31.521 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.634+0000 7fb59cbfe640 10 monclient: get_monmap_and_config 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.634+0000 7fb59cbfe640 10 monclient: build_initial_monmap 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.634+0000 7fb59cbfe640 10 monclient: monmap: 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:27:31.636223+0000 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:27:31.636223+0000 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:31.636 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:31.637 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:31.637 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:27:31.637 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:27:31.637 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:31.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.635+0000 7fb59cbfe640 1 Processor -- start 2024-03-25T18:27:31.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.635+0000 7fb59cbfe640 1 -- start start 2024-03-25T18:27:31.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.635+0000 7fb59cbfe640 10 monclient: init 2024-03-25T18:27:31.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:31.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:31.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 10 monclient(hunting): picked mon.noname-c con 0x7fb5980a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:31.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 10 monclient(hunting): picked mon.noname-a con 0x7fb59813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:31.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 10 monclient(hunting): picked mon.noname-b con 0x7fb5981430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:31.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb598142520 con 0x7fb59813f710 2024-03-25T18:27:31.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5980b28a0 con 0x7fb5980a98f0 2024-03-25T18:27:31.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5980b2a30 con 0x7fb5981430c0 2024-03-25T18:27:31.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:31.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.636+0000 7fb59cbfe640 10 monclient(hunting): authenticate will time out at 7482.925525s 2024-03-25T18:27:31.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.651+0000 7fb59659c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb5980a98f0 0x7fb5980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:52340/0 (socket says 172.21.15.102:52340) 2024-03-25T18:27:31.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.651+0000 7fb59659c640 1 -- 172.21.15.102:0/985740695 learned_addr learned my addr 172.21.15.102:0/985740695 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:31.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.658+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954058821 0 0) 0x7fb598142520 con 0x7fb59813f710 2024-03-25T18:27:31.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.658+0000 7fb59559a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.658+0000 7fb59559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.658+0000 7fb59559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.658+0000 7fb59559a640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:27:31.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.658+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb580003520 con 0x7fb59813f710 2024-03-25T18:27:31.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.660+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795651296 0 0) 0x7fb5980b28a0 con 0x7fb5980a98f0 2024-03-25T18:27:31.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.660+0000 7fb59559a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.660+0000 7fb59559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.660+0000 7fb59559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.660+0000 7fb59559a640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:27:31.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.660+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598142520 con 0x7fb5980a98f0 2024-03-25T18:27:31.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.667+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577765947 0 0) 0x7fb5980b2a30 con 0x7fb5981430c0 2024-03-25T18:27:31.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.667+0000 7fb59559a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.667+0000 7fb59559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.667+0000 7fb59559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.667+0000 7fb59559a640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:27:31.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.667+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5980b28a0 con 0x7fb5981430c0 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1729202231 0 0) 0x7fb598142520 con 0x7fb5980a98f0 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 >> v1:172.21.15.102:6790/0 conn(0x7fb5981430c0 legacy=0x7fb59814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 >> v1:172.21.15.102:6789/0 conn(0x7fb59813f710 legacy=0x7fb5981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 10 monclient: _finish_hunting 0 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 1 monclient: found mon.noname-c 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5980b44c0 con 0x7fb5980a98f0 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 10 monclient: _finish_auth 0 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 10 monclient: _check_auth_tickets 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:31.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb588001670 con 0x7fb5980a98f0 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59cbfe640 5 monclient: authenticate success, global_id 4297 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59559a640 10 monclient: dump: 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:31.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.668+0000 7fb59cbfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:31.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.692+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5880017d0 con 0x7fb5980a98f0 2024-03-25T18:27:31.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.693+0000 7fb59559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.693+0000 7fb59559a640 1 -- 172.21.15.102:0/985740695 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb588001be0 con 0x7fb5980a98f0 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.693+0000 7fb59559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.693+0000 7fb59559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.693+0000 7fb59559a640 10 monclient: dump: 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:31.695 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:31.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.693+0000 7fb59cbfe640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:31.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.693+0000 7fb59cbfe640 4 set_mon_vals no callback set 2024-03-25T18:27:31.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.694+0000 7fb59cbfe640 10 monclient: shutdown 2024-03-25T18:27:31.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.694+0000 7fb59cbfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:31.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.694+0000 7fb59cbfe640 1 -- 172.21.15.102:0/985740695 >> v1:172.21.15.146:6789/0 conn(0x7fb5980a98f0 legacy=0x7fb5980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.694+0000 7fb59cbfe640 1 -- 172.21.15.102:0/985740695 shutdown_connections 2024-03-25T18:27:31.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.694+0000 7fb59cbfe640 1 -- 172.21.15.102:0/985740695 >> 172.21.15.102:0/985740695 conn(0x7fb5980b7ea0 msgr2=0x7fb5980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.694+0000 7fb59cbfe640 1 -- 172.21.15.102:0/985740695 shutdown_connections 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.694+0000 7fb59cbfe640 1 -- 172.21.15.102:0/985740695 wait complete. 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.695+0000 7fb59cbfe640 10 monclient: build_initial_monmap 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.695+0000 7fb59cbfe640 10 monclient: monmap: 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:31.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.696+0000 7fb59cbfe640 1 librados: starting msgr at 2024-03-25T18:27:31.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.696+0000 7fb59cbfe640 1 librados: starting objecter 2024-03-25T18:27:31.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.696+0000 7fb59cbfe640 1 Processor -- start 2024-03-25T18:27:31.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.696+0000 7fb59cbfe640 1 -- start start 2024-03-25T18:27:31.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.696+0000 7fb59cbfe640 1 librados: setting wanted keys 2024-03-25T18:27:31.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.696+0000 7fb59cbfe640 1 librados: calling monclient init 2024-03-25T18:27:31.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.696+0000 7fb59cbfe640 10 monclient: init 2024-03-25T18:27:31.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:31.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:31.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 10 monclient(hunting): picked mon.noname-c con 0x7fb5980a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:31.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 10 monclient(hunting): picked mon.noname-b con 0x7fb59813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:31.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 10 monclient(hunting): picked mon.noname-a con 0x7fb5981430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:31.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb598137f50 con 0x7fb5981430c0 2024-03-25T18:27:31.701 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb598139830 con 0x7fb59813f710 2024-03-25T18:27:31.701 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5980aba50 con 0x7fb5980a98f0 2024-03-25T18:27:31.701 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:31.701 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.697+0000 7fb59cbfe640 10 monclient(hunting): authenticate will time out at 7482.986608s 2024-03-25T18:27:31.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.706+0000 7fb596d9d640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb5981430c0 0x7fb598136240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46376/0 (socket says 172.21.15.102:46376) 2024-03-25T18:27:31.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.706+0000 7fb596d9d640 1 -- 172.21.15.102:0/3534292265 learned_addr learned my addr 172.21.15.102:0/3534292265 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:31.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.708+0000 7fb586ffd640 10 client.?.objecter ms_handle_connect 0x7fb5981430c0 2024-03-25T18:27:31.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.708+0000 7fb586ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:31.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.711+0000 7fb586ffd640 10 client.?.objecter ms_handle_connect 0x7fb5980a98f0 2024-03-25T18:27:31.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.711+0000 7fb586ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:31.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.712+0000 7fb586ffd640 10 client.?.objecter ms_handle_connect 0x7fb59813f710 2024-03-25T18:27:31.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.712+0000 7fb586ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:31.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.720+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231477574 0 0) 0x7fb5980aba50 con 0x7fb5980a98f0 2024-03-25T18:27:31.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.720+0000 7fb586ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.720+0000 7fb586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.720+0000 7fb586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.720+0000 7fb586ffd640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:27:31.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.720+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb578004eb0 con 0x7fb5980a98f0 2024-03-25T18:27:31.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.722+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377039818 0 0) 0x7fb598137f50 con 0x7fb5981430c0 2024-03-25T18:27:31.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.722+0000 7fb586ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.722+0000 7fb586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.722+0000 7fb586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.722+0000 7fb586ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:27:31.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.722+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5980aba50 con 0x7fb5981430c0 2024-03-25T18:27:31.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.728+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837934031 0 0) 0x7fb598139830 con 0x7fb59813f710 2024-03-25T18:27:31.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.728+0000 7fb586ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:31.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.728+0000 7fb586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:31.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.728+0000 7fb586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:31.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.728+0000 7fb586ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:27:31.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.728+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598137f50 con 0x7fb59813f710 2024-03-25T18:27:31.731 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.729+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 349975893 0 0) 0x7fb578004eb0 con 0x7fb5980a98f0 2024-03-25T18:27:31.731 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.729+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb598139830 con 0x7fb5980a98f0 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.739+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2483499406 0 0) 0x7fb598137f50 con 0x7fb59813f710 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.739+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb578004eb0 con 0x7fb59813f710 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.739+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb58c0017b0 con 0x7fb59813f710 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.739+0000 7fb586ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.739+0000 7fb586ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.739+0000 7fb586ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:31.741 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:31.742 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:31.742 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:31.742 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:31.742 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:31.742 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:31.742 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:31.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.739+0000 7fb586ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:31.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb588001be0 con 0x7fb5980a98f0 2024-03-25T18:27:31.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:31.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:31.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:31.746 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2108449463 0 0) 0x7fb598139830 con 0x7fb5980a98f0 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 >> v1:172.21.15.146:6789/0 conn(0x7fb59813f710 legacy=0x7fb598135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 >> v1:172.21.15.102:6789/0 conn(0x7fb5981430c0 legacy=0x7fb598136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:31.747 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.743+0000 7fb586ffd640 1 monclient: found mon.c 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb586ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5980abd30 con 0x7fb5980a98f0 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb586ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb586ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 5 monclient: authenticate success, global_id 4286 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 10 monclient: _renew_subs 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb598137a00 con 0x7fb5980a98f0 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 10 monclient: _renew_subs 2024-03-25T18:27:31.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:31.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb598134970 con 0x7fb5980a98f0 2024-03-25T18:27:31.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 1 librados: init done 2024-03-25T18:27:31.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.744+0000 7fb59cbfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:31.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.745+0000 7fb59cbfe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:31.750 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.745+0000 7fb59cbfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:31.750 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:31.745+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5980627b0 con 0x7fb5980a98f0 2024-03-25T18:27:32.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.048+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb588004a90 con 0x7fb5980a98f0 2024-03-25T18:27:32.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.048+0000 7fb586ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:32.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.048+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb588004da0 con 0x7fb5980a98f0 2024-03-25T18:27:32.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.048+0000 7fb586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:32.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.048+0000 7fb586ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:32.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.048+0000 7fb586ffd640 10 monclient: dump: 2024-03-25T18:27:32.051 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:32.051 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:32.052 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:32.052 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:32.052 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:32.052 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:32.052 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:32.052 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:32.052 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:32.053 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:32.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.048+0000 7fb594d99640 4 set_mon_vals no callback set 2024-03-25T18:27:32.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.049+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fb58801ad70 con 0x7fb5980a98f0 2024-03-25T18:27:32.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.050+0000 7fb595d9b640 10 monclient: get_auth_request con 0x7fb57806dfc0 auth_method 0 2024-03-25T18:27:32.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.050+0000 7fb586ffd640 10 client.4286.objecter ms_handle_connect 0x7fb57806dfc0 2024-03-25T18:27:32.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.055+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fb588054930 con 0x7fb5980a98f0 2024-03-25T18:27:32.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.055+0000 7fb586ffd640 10 client.4286.objecter ms_dispatch 0x7fb598000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:32.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.055+0000 7fb586ffd640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:32.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.055+0000 7fb586ffd640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:32.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.055+0000 7fb586ffd640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-25T18:27:32.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.055+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb588050270 con 0x7fb5980a98f0 2024-03-25T18:27:32.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.055+0000 7fb586ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:32.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.055+0000 7fb586ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.292+0000 7fb59cbfe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.292+0000 7fb59cbfe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.292+0000 7fb59cbfe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:32.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.292+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5980b71a0 con 0x7fb5980a98f0 2024-03-25T18:27:32.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.358+0000 7fb586ffd640 1 -- 172.21.15.102:0/3534292265 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fb588052370 con 0x7fb5980a98f0 2024-03-25T18:27:32.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.358+0000 7fb586ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:32.360 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.358+0000 7fb586ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:32.360 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:27:32.360 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:32.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 10 librados: watch_flush enter 2024-03-25T18:27:32.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 10 librados: watch_flush exit 2024-03-25T18:27:32.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-25T18:27:32.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 10 client.4286.objecter successfully canceled tick 2024-03-25T18:27:32.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 >> v1:172.21.15.146:6800/3848252714 conn(0x7fb57806dfc0 legacy=0x7fb578070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 10 monclient: shutdown 2024-03-25T18:27:32.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:32.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 >> v1:172.21.15.102:6790/0 conn(0x7fb5980a98f0 legacy=0x7fb598134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 shutdown_connections 2024-03-25T18:27:32.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 >> 172.21.15.102:0/3534292265 conn(0x7fb5980b7ea0 msgr2=0x7fb598074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:32.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 shutdown_connections 2024-03-25T18:27:32.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 1 -- 172.21.15.102:0/3534292265 wait complete. 2024-03-25T18:27:32.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:32.360+0000 7fb59cbfe640 1 librados: shutdown 2024-03-25T18:27:32.372 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:38.007 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:54.243 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.060 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:54.296 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.089 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:01:54.325 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.122 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:54.358 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.124 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:54.338 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.129 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:54.342 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.135 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:54.371 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.169 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:01:54.405 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.171 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:01:54.385 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:39.178 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:39.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.285+0000 7f4090cfb640 10 monclient: get_monmap_and_config 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.285+0000 7f4090cfb640 10 monclient: build_initial_monmap 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.285+0000 7f4090cfb640 10 monclient: monmap: 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:27:39.286993+0000 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:27:39.286993+0000 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:27:39.287 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:27:39.288 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:39.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 1 Processor -- start 2024-03-25T18:27:39.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 1 -- start start 2024-03-25T18:27:39.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 10 monclient: init 2024-03-25T18:27:39.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:39.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:39.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f408c0b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:39.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f408c0b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:39.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f408c14be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:39.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c0af940 con 0x7f408c0b4d60 2024-03-25T18:27:39.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c0b0a70 con 0x7f408c0b1730 2024-03-25T18:27:39.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c0b0d50 con 0x7f408c14be90 2024-03-25T18:27:39.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:39.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.286+0000 7f4090cfb640 10 monclient(hunting): authenticate will time out at 7490.575692s 2024-03-25T18:27:39.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.316+0000 7f4083fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f408c0b1730 0x7f408c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:57074/0 (socket says 172.21.15.102:57074) 2024-03-25T18:27:39.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.316+0000 7f4083fff640 1 -- 172.21.15.102:0/625577781 learned_addr learned my addr 172.21.15.102:0/625577781 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:39.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.334+0000 7f408ad9d640 1 -- 172.21.15.102:0/625577781 >> v1:172.21.15.102:6790/0 conn(0x7f408c14be90 legacy=0x7f408c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T18:27:39.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.334+0000 7f408ad9d640 1 -- 172.21.15.102:0/625577781 >> v1:172.21.15.102:6790/0 conn(0x7f408c14be90 legacy=0x7f408c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:27:39.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.334+0000 7f408ad9d640 1 --1- 172.21.15.102:0/625577781 >> v1:172.21.15.102:6790/0 conn(0x7f408c14be90 0x7f408c14e280 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:27:39.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.336+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136328755 0 0) 0x7f408c0b0a70 con 0x7f408c0b1730 2024-03-25T18:27:39.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.336+0000 7f4089d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.336+0000 7f4089d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.336+0000 7f4089d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.336+0000 7f4089d9b640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:27:39.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.336+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4078003520 con 0x7f408c0b1730 2024-03-25T18:27:39.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.343+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465676714 0 0) 0x7f408c0af940 con 0x7f408c0b4d60 2024-03-25T18:27:39.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.343+0000 7f4089d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.343+0000 7f4089d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.343+0000 7f4089d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.343+0000 7f4089d9b640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:27:39.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.343+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f408c0b0a70 con 0x7f408c0b4d60 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 963357997 0 0) 0x7f4078003520 con 0x7f408c0b1730 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 >> v1:172.21.15.102:6790/0 conn(0x7f408c14be90 legacy=0x7f408c14e280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 >> v1:172.21.15.102:6789/0 conn(0x7f408c0b4d60 legacy=0x7f408c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 1 monclient: found mon.noname-c 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c0b1030 con 0x7f408c0b1730 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 10 monclient: _finish_auth 0 2024-03-25T18:27:39.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4074001690 con 0x7f408c0b1730 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4090cfb640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4090cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4089d9b640 10 monclient: dump: 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:39.352 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:39.353 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:39.353 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:39.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.349+0000 7f4090cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:39.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.363+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40740017f0 con 0x7f408c0b1730 2024-03-25T18:27:39.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.363+0000 7f4089d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:39.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.363+0000 7f4089d9b640 1 -- 172.21.15.102:0/625577781 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4074001c00 con 0x7f408c0b1730 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.363+0000 7f4089d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.364+0000 7f4089d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.364+0000 7f4089d9b640 10 monclient: dump: 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:39.366 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:39.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.364+0000 7f4090cfb640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:39.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.364+0000 7f4090cfb640 4 set_mon_vals no callback set 2024-03-25T18:27:39.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.364+0000 7f4090cfb640 10 monclient: shutdown 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.364+0000 7f4090cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.364+0000 7f4090cfb640 1 -- 172.21.15.102:0/625577781 >> v1:172.21.15.146:6789/0 conn(0x7f408c0b1730 legacy=0x7f408c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.365+0000 7f4090cfb640 1 -- 172.21.15.102:0/625577781 shutdown_connections 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.365+0000 7f4090cfb640 1 -- 172.21.15.102:0/625577781 >> 172.21.15.102:0/625577781 conn(0x7f408c0617a0 msgr2=0x7f408c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.365+0000 7f4090cfb640 1 -- 172.21.15.102:0/625577781 shutdown_connections 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.365+0000 7f4090cfb640 1 -- 172.21.15.102:0/625577781 wait complete. 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.366+0000 7f4090cfb640 10 monclient: build_initial_monmap 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.366+0000 7f4090cfb640 10 monclient: monmap: 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:27:39.368 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:39.369 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:27:39.369 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:39.369 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:27:39.369 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:27:39.369 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:39.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.366+0000 7f4090cfb640 1 librados: starting msgr at 2024-03-25T18:27:39.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.366+0000 7f4090cfb640 1 librados: starting objecter 2024-03-25T18:27:39.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 1 Processor -- start 2024-03-25T18:27:39.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 1 -- start start 2024-03-25T18:27:39.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 1 librados: setting wanted keys 2024-03-25T18:27:39.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 1 librados: calling monclient init 2024-03-25T18:27:39.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 10 monclient: init 2024-03-25T18:27:39.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:39.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:39.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 10 monclient(hunting): picked mon.noname-b con 0x7f408c0b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:39.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.367+0000 7f4090cfb640 10 monclient(hunting): picked mon.noname-c con 0x7f408c0b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:39.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.368+0000 7f4090cfb640 10 monclient(hunting): picked mon.noname-a con 0x7f408c14be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:39.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.368+0000 7f4090cfb640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c13a7d0 con 0x7f408c14be90 2024-03-25T18:27:39.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.368+0000 7f4090cfb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c13b100 con 0x7f408c0b1730 2024-03-25T18:27:39.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.368+0000 7f4090cfb640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f408c142b10 con 0x7f408c0b4d60 2024-03-25T18:27:39.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.368+0000 7f4090cfb640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:39.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.368+0000 7f4090cfb640 10 monclient(hunting): authenticate will time out at 7490.657054s 2024-03-25T18:27:39.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.377+0000 7f408a59c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f408c0b1730 0x7f408c139290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:57124/0 (socket says 172.21.15.102:57124) 2024-03-25T18:27:39.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.377+0000 7f408a59c640 1 -- 172.21.15.102:0/556946223 learned_addr learned my addr 172.21.15.102:0/556946223 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:39.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.379+0000 7f4082ffd640 10 client.?.objecter ms_handle_connect 0x7f408c0b1730 2024-03-25T18:27:39.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.379+0000 7f4082ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:39.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.380+0000 7f4082ffd640 10 client.?.objecter ms_handle_connect 0x7f408c14be90 2024-03-25T18:27:39.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.380+0000 7f4082ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:39.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.383+0000 7f4082ffd640 10 client.?.objecter ms_handle_connect 0x7f408c0b4d60 2024-03-25T18:27:39.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.383+0000 7f4082ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:39.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.392+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603669842 0 0) 0x7f408c142b10 con 0x7f408c0b4d60 2024-03-25T18:27:39.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.392+0000 7f4082ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.392+0000 7f4082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.392+0000 7f4082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.392+0000 7f4082ffd640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:27:39.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.392+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c004eb0 con 0x7f408c0b4d60 2024-03-25T18:27:39.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.394+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319054364 0 0) 0x7f408c13b100 con 0x7f408c0b1730 2024-03-25T18:27:39.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.394+0000 7f4082ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.394+0000 7f4082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.394+0000 7f4082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.394+0000 7f4082ffd640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:27:39.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.394+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f408c142b10 con 0x7f408c0b1730 2024-03-25T18:27:39.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.403+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175371868 0 0) 0x7f408c13a7d0 con 0x7f408c14be90 2024-03-25T18:27:39.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.403+0000 7f4082ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:39.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.403+0000 7f4082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:39.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.403+0000 7f4082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:39.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.403+0000 7f4082ffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:27:39.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.403+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f408c13b100 con 0x7f408c14be90 2024-03-25T18:27:39.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.408+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2722185043 0 0) 0x7f408c142b10 con 0x7f408c0b1730 2024-03-25T18:27:39.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.408+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f408c13a7d0 con 0x7f408c0b1730 2024-03-25T18:27:39.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.408+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f407c0019f0 con 0x7f408c0b1730 2024-03-25T18:27:39.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.409+0000 7f4082ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.409+0000 7f4082ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:39.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.409+0000 7f4082ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:39.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.409+0000 7f4082ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.413+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3139300 0 0) 0x7f406c004eb0 con 0x7f408c0b4d60 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.413+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f408c142b10 con 0x7f408c0b4d60 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.413+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4074001bc0 con 0x7f408c0b4d60 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.413+0000 7f4082ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.413+0000 7f4082ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.413+0000 7f4082ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.415 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.416 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.416 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:39.416 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:39.416 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:39.416 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:39.416 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:39.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.415+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3630783519 0 0) 0x7f408c13a7d0 con 0x7f408c0b1730 2024-03-25T18:27:39.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.415+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 >> v1:172.21.15.102:6790/0 conn(0x7f408c0b4d60 legacy=0x7f408c13b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 >> v1:172.21.15.102:6789/0 conn(0x7f408c14be90 legacy=0x7f408c13bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:39.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:39.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 1 monclient: found mon.b 2024-03-25T18:27:39.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c143360 con 0x7f408c0b1730 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 10 monclient: _renew_subs 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c1435b0 con 0x7f408c0b1730 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 10 monclient: _renew_subs 2024-03-25T18:27:39.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:39.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c1e32f0 con 0x7f408c0b1730 2024-03-25T18:27:39.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.416+0000 7f4090cfb640 1 librados: init done 2024-03-25T18:27:39.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.417+0000 7f4090cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:39.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.417+0000 7f4090cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:39.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.417+0000 7f4090cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:39.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.417+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c0627b0 con 0x7f408c0b1730 2024-03-25T18:27:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.712+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f407c003610 con 0x7f408c0b1730 2024-03-25T18:27:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.712+0000 7f4082ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.712+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f407c003800 con 0x7f408c0b1730 2024-03-25T18:27:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.712+0000 7f4082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.712+0000 7f4082ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.712+0000 7f4082ffd640 10 monclient: dump: 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:39.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.712+0000 7f408959a640 4 set_mon_vals no callback set 2024-03-25T18:27:39.716 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.713+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f407c003a20 con 0x7f408c0b1730 2024-03-25T18:27:39.716 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.714+0000 7f4083fff640 10 monclient: get_auth_request con 0x7f406c06df70 auth_method 0 2024-03-25T18:27:39.716 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.714+0000 7f4082ffd640 10 client.4306.objecter ms_handle_connect 0x7f406c06df70 2024-03-25T18:27:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.719+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f407c0546c0 con 0x7f408c0b1730 2024-03-25T18:27:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.719+0000 7f4082ffd640 10 client.4306.objecter ms_dispatch 0x7f408c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.719+0000 7f4082ffd640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.719+0000 7f4082ffd640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.719+0000 7f4082ffd640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T18:27:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.719+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f407c0501d0 con 0x7f408c0b1730 2024-03-25T18:27:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.719+0000 7f4082ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.719+0000 7f4082ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:39.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.966+0000 7f4090cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:39.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.966+0000 7f4090cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:39.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.966+0000 7f4090cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:27:39.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:39.966+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f408c069f40 con 0x7f408c0b1730 2024-03-25T18:27:40.007 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.005+0000 7f4082ffd640 1 -- 172.21.15.102:0/556946223 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f407c052100 con 0x7f408c0b1730 2024-03-25T18:27:40.007 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.005+0000 7f4082ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:40.007 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.005+0000 7f4082ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:40.007 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:27:40.008 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:40.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 10 librados: watch_flush enter 2024-03-25T18:27:40.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 10 librados: watch_flush exit 2024-03-25T18:27:40.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T18:27:40.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 10 client.4306.objecter successfully canceled tick 2024-03-25T18:27:40.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 >> v1:172.21.15.146:6800/3848252714 conn(0x7f406c06df70 legacy=0x7f406c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 10 monclient: shutdown 2024-03-25T18:27:40.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:40.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 >> v1:172.21.15.146:6789/0 conn(0x7f408c0b1730 legacy=0x7f408c139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 shutdown_connections 2024-03-25T18:27:40.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 >> 172.21.15.102:0/556946223 conn(0x7f408c0617a0 msgr2=0x7f408c137380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:40.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 shutdown_connections 2024-03-25T18:27:40.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 1 -- 172.21.15.102:0/556946223 wait complete. 2024-03-25T18:27:40.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:40.007+0000 7f4090cfb640 1 librados: shutdown 2024-03-25T18:27:40.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:46.828 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:46.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.947+0000 7f7a0ff1c640 10 monclient: get_monmap_and_config 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.947+0000 7f7a0ff1c640 10 monclient: build_initial_monmap 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.947+0000 7f7a0ff1c640 10 monclient: monmap: 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:27:46.947805+0000 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:27:46.947805+0000 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:27:46.948 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.947+0000 7f7a0ff1c640 1 Processor -- start 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 1 -- start start 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 10 monclient: init 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 10 monclient(hunting): picked mon.noname-c con 0x7f7a080a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 10 monclient(hunting): picked mon.noname-b con 0x7f7a0813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 10 monclient(hunting): picked mon.noname-a con 0x7f7a081430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a08142520 con 0x7f7a081430c0 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a080b28a0 con 0x7f7a080a98f0 2024-03-25T18:27:46.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a080b2a30 con 0x7f7a0813f710 2024-03-25T18:27:46.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:46.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.948+0000 7f7a0ff1c640 10 monclient(hunting): authenticate will time out at 7498.236708s 2024-03-25T18:27:46.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.954+0000 7f7a0d4b7640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7a0813f710 0x7f7a081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:45524/0 (socket says 172.21.15.102:45524) 2024-03-25T18:27:46.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.954+0000 7f7a0d4b7640 1 -- 172.21.15.102:0/1549358911 learned_addr learned my addr 172.21.15.102:0/1549358911 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:46.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.967+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244840973 0 0) 0x7f7a080b28a0 con 0x7f7a080a98f0 2024-03-25T18:27:46.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.967+0000 7f7a0ccb6640 10 monclient(hunting): _negotiate 2024-03-25T18:27:46.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.967+0000 7f7a0ccb6640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:46.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.967+0000 7f7a0ccb6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:46.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.967+0000 7f7a0ccb6640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:27:46.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.967+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79f4003520 con 0x7f7a080a98f0 2024-03-25T18:27:46.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.972+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669435391 0 0) 0x7f7a08142520 con 0x7f7a081430c0 2024-03-25T18:27:46.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.973+0000 7f7a0ccb6640 10 monclient(hunting): _negotiate 2024-03-25T18:27:46.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.973+0000 7f7a0ccb6640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:46.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.973+0000 7f7a0ccb6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:46.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.973+0000 7f7a0ccb6640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:27:46.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.973+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a080b28a0 con 0x7f7a081430c0 2024-03-25T18:27:46.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2313718600 0 0) 0x7f79f4003520 con 0x7f7a080a98f0 2024-03-25T18:27:46.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 >> v1:172.21.15.102:6790/0 conn(0x7f7a0813f710 legacy=0x7f7a081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:46.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 >> v1:172.21.15.102:6789/0 conn(0x7f7a081430c0 legacy=0x7f7a0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 10 monclient: _finish_hunting 0 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 1 monclient: found mon.noname-c 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a080b44c0 con 0x7f7a080a98f0 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 10 monclient: _finish_auth 0 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 10 monclient: _check_auth_tickets 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7a04001690 con 0x7f7a080a98f0 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.975+0000 7f7a0ccb6640 10 monclient: dump: 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:46.977 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.976+0000 7f7a0ff1c640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:27:46.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.976+0000 7f7a0ff1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.981+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a040017f0 con 0x7f7a080a98f0 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.981+0000 7f7a0ccb6640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.981+0000 7f7a0ccb6640 1 -- 172.21.15.102:0/1549358911 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7a04001c00 con 0x7f7a080a98f0 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.981+0000 7f7a0ccb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.981+0000 7f7a0ccb6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.981+0000 7f7a0ccb6640 10 monclient: dump: 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:46.982 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:46.983 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:46.983 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:46.983 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:46.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.981+0000 7f7a0ff1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:46.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.981+0000 7f7a0ff1c640 4 set_mon_vals no callback set 2024-03-25T18:27:46.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.982+0000 7f7a0ff1c640 10 monclient: shutdown 2024-03-25T18:27:46.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.982+0000 7f7a0ff1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:46.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.982+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/1549358911 >> v1:172.21.15.146:6789/0 conn(0x7f7a080a98f0 legacy=0x7f7a080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:46.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.982+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/1549358911 shutdown_connections 2024-03-25T18:27:46.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.982+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/1549358911 >> 172.21.15.102:0/1549358911 conn(0x7f7a080b7ea0 msgr2=0x7f7a080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:46.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.983+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/1549358911 shutdown_connections 2024-03-25T18:27:46.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.983+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/1549358911 wait complete. 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 10 monclient: build_initial_monmap 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 10 monclient: monmap: 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:46.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 1 librados: starting msgr at 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 1 librados: starting objecter 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 1 Processor -- start 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 1 -- start start 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 1 librados: setting wanted keys 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 1 librados: calling monclient init 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.984+0000 7f7a0ff1c640 10 monclient: init 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 10 monclient(hunting): picked mon.noname-c con 0x7f7a080a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 10 monclient(hunting): picked mon.noname-a con 0x7f7a0813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 10 monclient(hunting): picked mon.noname-b con 0x7f7a081430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a08138250 con 0x7f7a0813f710 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a08137d50 con 0x7f7a081430c0 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a08137ee0 con 0x7f7a080a98f0 2024-03-25T18:27:46.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:46.988 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.985+0000 7f7a0ff1c640 10 monclient(hunting): authenticate will time out at 7498.273359s 2024-03-25T18:27:46.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.995+0000 7f7a0dcb8640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7a080a98f0 0x7f7a08137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:45540/0 (socket says 172.21.15.102:45540) 2024-03-25T18:27:46.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.995+0000 7f7a0dcb8640 1 -- 172.21.15.102:0/685115196 learned_addr learned my addr 172.21.15.102:0/685115196 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:47.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.999+0000 7f79fe7fc640 10 client.?.objecter ms_handle_connect 0x7f7a080a98f0 2024-03-25T18:27:47.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:46.999+0000 7f79fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:47.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.000+0000 7f79fe7fc640 10 client.?.objecter ms_handle_connect 0x7f7a0813f710 2024-03-25T18:27:47.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.000+0000 7f79fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:47.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.006+0000 7f79fe7fc640 10 client.?.objecter ms_handle_connect 0x7f7a081430c0 2024-03-25T18:27:47.007 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.006+0000 7f79fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:47.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.007+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699095875 0 0) 0x7f7a08137ee0 con 0x7f7a080a98f0 2024-03-25T18:27:47.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.007+0000 7f79fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.007+0000 7f79fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.007+0000 7f79fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.007+0000 7f79fe7fc640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:27:47.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.008+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ec004eb0 con 0x7f7a080a98f0 2024-03-25T18:27:47.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.009+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077184218 0 0) 0x7f7a08138250 con 0x7f7a0813f710 2024-03-25T18:27:47.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.009+0000 7f79fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.009+0000 7f79fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.009+0000 7f79fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.009+0000 7f79fe7fc640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:27:47.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.009+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a08137ee0 con 0x7f7a0813f710 2024-03-25T18:27:47.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.020+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439445057 0 0) 0x7f7a08137d50 con 0x7f7a081430c0 2024-03-25T18:27:47.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.020+0000 7f79fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.020+0000 7f79fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.020+0000 7f79fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.020+0000 7f79fe7fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:27:47.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.020+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a08138250 con 0x7f7a081430c0 2024-03-25T18:27:47.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.021+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658990483 0 0) 0x7f79ec004eb0 con 0x7f7a080a98f0 2024-03-25T18:27:47.024 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.021+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a08137d50 con 0x7f7a080a98f0 2024-03-25T18:27:47.024 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.021+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7a04001690 con 0x7f7a080a98f0 2024-03-25T18:27:47.024 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.021+0000 7f79fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.024 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.021+0000 7f79fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:47.024 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.021+0000 7f79fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:47.024 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:47.025 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:47.025 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:47.025 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:47.025 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.025 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:47.026 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:47.026 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:47.026 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:47.026 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:47.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.021+0000 7f79fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:47.027 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.022+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654780367 0 0) 0x7f7a08137ee0 con 0x7f7a0813f710 2024-03-25T18:27:47.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.022+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79ec004eb0 con 0x7f7a0813f710 2024-03-25T18:27:47.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.022+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f79f80018a0 con 0x7f7a0813f710 2024-03-25T18:27:47.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.022+0000 7f79fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.022+0000 7f79fe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:47.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.022+0000 7f79fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:47.028 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:47.029 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:47.029 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:47.029 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:47.029 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.029 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:47.029 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:47.029 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:47.030 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:47.030 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:47.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.030+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3687393285 0 0) 0x7f7a08138250 con 0x7f7a081430c0 2024-03-25T18:27:47.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.031+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a08137ee0 con 0x7f7a081430c0 2024-03-25T18:27:47.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.031+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7a00001bd0 con 0x7f7a081430c0 2024-03-25T18:27:47.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.031+0000 7f79fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.031+0000 7f79fe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:47.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.031+0000 7f79fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:47.032 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:47.033 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:47.033 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:47.033 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:47.033 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.033 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:47.033 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:47.033 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:47.034 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:47.034 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:47.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3900564995 0 0) 0x7f7a08137d50 con 0x7f7a080a98f0 2024-03-25T18:27:47.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 >> v1:172.21.15.146:6789/0 conn(0x7f7a081430c0 legacy=0x7f7a081d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 >> v1:172.21.15.102:6789/0 conn(0x7f7a0813f710 legacy=0x7f7a081d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:47.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 1 monclient: found mon.c 2024-03-25T18:27:47.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a080b7730 con 0x7f7a080a98f0 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f79fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 5 monclient: authenticate success, global_id 4295 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 10 monclient: _renew_subs 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a080b7920 con 0x7f7a080a98f0 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 10 monclient: _renew_subs 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a081e33d0 con 0x7f7a080a98f0 2024-03-25T18:27:47.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.034+0000 7f7a0ff1c640 1 librados: init done 2024-03-25T18:27:47.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.035+0000 7f7a0ff1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:47.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.035+0000 7f7a0ff1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:47.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.035+0000 7f7a0ff1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:47.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.035+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a080627b0 con 0x7f7a080a98f0 2024-03-25T18:27:47.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.332+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a040038a0 con 0x7f7a080a98f0 2024-03-25T18:27:47.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.332+0000 7f79fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.332+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7a04001cc0 con 0x7f7a080a98f0 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.332+0000 7f79fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.332+0000 7f79fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.332+0000 7f79fe7fc640 10 monclient: dump: 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:47.334 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:47.335 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:47.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.332+0000 7f79fffff640 4 set_mon_vals no callback set 2024-03-25T18:27:47.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.333+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f7a04003db0 con 0x7f7a080a98f0 2024-03-25T18:27:47.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.334+0000 7f7a0d4b7640 10 monclient: get_auth_request con 0x7f79ec06e110 auth_method 0 2024-03-25T18:27:47.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.334+0000 7f79fe7fc640 10 client.4295.objecter ms_handle_connect 0x7f79ec06e110 2024-03-25T18:27:47.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.340+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f7a04054890 con 0x7f7a080a98f0 2024-03-25T18:27:47.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.340+0000 7f79fe7fc640 10 client.4295.objecter ms_dispatch 0x7f7a08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:47.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.340+0000 7f79fe7fc640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:47.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.340+0000 7f79fe7fc640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:47.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.340+0000 7f79fe7fc640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-25T18:27:47.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.340+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a040503a0 con 0x7f7a080a98f0 2024-03-25T18:27:47.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.340+0000 7f79fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:47.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.340+0000 7f79fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:47.550 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.549+0000 7f7a0ff1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:47.550 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.549+0000 7f7a0ff1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:47.550 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.549+0000 7f7a0ff1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:27:47.550 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.549+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a0813df50 con 0x7f7a080a98f0 2024-03-25T18:27:47.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.586+0000 7f79fe7fc640 1 -- 172.21.15.102:0/685115196 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f7a040522d0 con 0x7f7a080a98f0 2024-03-25T18:27:47.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.586+0000 7f79fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:47.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.586+0000 7f79fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:47.587 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:27:47.587 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:47.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.588+0000 7f7a0ff1c640 10 librados: watch_flush enter 2024-03-25T18:27:47.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.588+0000 7f7a0ff1c640 10 librados: watch_flush exit 2024-03-25T18:27:47.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.588+0000 7f7a0ff1c640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-25T18:27:47.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.588+0000 7f7a0ff1c640 10 client.4295.objecter successfully canceled tick 2024-03-25T18:27:47.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 >> v1:172.21.15.146:6800/3848252714 conn(0x7f79ec06e110 legacy=0x7f79ec0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 10 monclient: shutdown 2024-03-25T18:27:47.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:47.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 >> v1:172.21.15.102:6790/0 conn(0x7f7a080a98f0 legacy=0x7f7a08137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:47.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 shutdown_connections 2024-03-25T18:27:47.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 >> 172.21.15.102:0/685115196 conn(0x7f7a080b7ea0 msgr2=0x7f7a08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:47.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 shutdown_connections 2024-03-25T18:27:47.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 1 -- 172.21.15.102:0/685115196 wait complete. 2024-03-25T18:27:47.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:47.589+0000 7f7a0ff1c640 1 librados: shutdown 2024-03-25T18:27:47.601 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:54.409 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.524+0000 7ff7ce515640 10 monclient: get_monmap_and_config 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.524+0000 7ff7ce515640 10 monclient: build_initial_monmap 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.524+0000 7ff7ce515640 10 monclient: monmap: 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:27:54.525055+0000 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:27:54.525055+0000 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:54.525 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:27:54.526 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:27:54.526 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:54.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.525+0000 7ff7ce515640 1 Processor -- start 2024-03-25T18:27:54.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.525+0000 7ff7ce515640 1 -- start start 2024-03-25T18:27:54.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.525+0000 7ff7ce515640 10 monclient: init 2024-03-25T18:27:54.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.525+0000 7ff7ce515640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:54.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.525+0000 7ff7ce515640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:54.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.525+0000 7ff7ce515640 10 monclient(hunting): picked mon.noname-b con 0x7ff7c80b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:54.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.525+0000 7ff7ce515640 10 monclient(hunting): picked mon.noname-a con 0x7ff7c80b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:54.527 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.526+0000 7ff7ce515640 10 monclient(hunting): picked mon.noname-c con 0x7ff7c814be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:54.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.526+0000 7ff7ce515640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c80af940 con 0x7ff7c80b1730 2024-03-25T18:27:54.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.526+0000 7ff7ce515640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c80b0a70 con 0x7ff7c814be90 2024-03-25T18:27:54.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.526+0000 7ff7ce515640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c80b0d50 con 0x7ff7c80b4d60 2024-03-25T18:27:54.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.526+0000 7ff7ce515640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:54.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.526+0000 7ff7ce515640 10 monclient(hunting): authenticate will time out at 7505.814122s 2024-03-25T18:27:54.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.536+0000 7ff7ccab2640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff7c814be90 0x7ff7c814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:55730/0 (socket says 172.21.15.102:55730) 2024-03-25T18:27:54.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.536+0000 7ff7ccab2640 1 -- 172.21.15.102:0/2660169464 learned_addr learned my addr 172.21.15.102:0/2660169464 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:54.551 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.550+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308114370 0 0) 0x7ff7c80b0d50 con 0x7ff7c80b4d60 2024-03-25T18:27:54.551 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.550+0000 7ff7c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.550+0000 7ff7c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.550+0000 7ff7c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.550+0000 7ff7c6ffd640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.550+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b0003520 con 0x7ff7c80b4d60 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.551+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383268827 0 0) 0x7ff7c80b0a70 con 0x7ff7c814be90 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.551+0000 7ff7c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.551+0000 7ff7c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.551+0000 7ff7c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.551+0000 7ff7c6ffd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:27:54.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.551+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c80b0d50 con 0x7ff7c814be90 2024-03-25T18:27:54.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 262104369 0 0) 0x7ff7b0003520 con 0x7ff7c80b4d60 2024-03-25T18:27:54.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 >> v1:172.21.15.146:6789/0 conn(0x7ff7c814be90 legacy=0x7ff7c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 >> v1:172.21.15.102:6789/0 conn(0x7ff7c80b1730 legacy=0x7ff7c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:54.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 1 monclient: found mon.noname-b 2024-03-25T18:27:54.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c80b1030 con 0x7ff7c80b4d60 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff7b8001690 con 0x7ff7c80b4d60 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7ce515640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7ce515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7c6ffd640 10 monclient: dump: 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:54.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.559+0000 7ff7ce515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:54.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.573+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7b80017f0 con 0x7ff7c80b4d60 2024-03-25T18:27:54.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.573+0000 7ff7c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:54.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.573+0000 7ff7c6ffd640 1 -- 172.21.15.102:0/2660169464 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff7b8001c00 con 0x7ff7c80b4d60 2024-03-25T18:27:54.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.573+0000 7ff7c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.573+0000 7ff7c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:54.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.573+0000 7ff7c6ffd640 10 monclient: dump: 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.573+0000 7ff7ce515640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:54.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.573+0000 7ff7ce515640 4 set_mon_vals no callback set 2024-03-25T18:27:54.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.574+0000 7ff7ce515640 10 monclient: shutdown 2024-03-25T18:27:54.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.574+0000 7ff7ce515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.574+0000 7ff7ce515640 1 -- 172.21.15.102:0/2660169464 >> v1:172.21.15.102:6790/0 conn(0x7ff7c80b4d60 legacy=0x7ff7c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.574+0000 7ff7ce515640 1 -- 172.21.15.102:0/2660169464 shutdown_connections 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.574+0000 7ff7ce515640 1 -- 172.21.15.102:0/2660169464 >> 172.21.15.102:0/2660169464 conn(0x7ff7c80617a0 msgr2=0x7ff7c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.574+0000 7ff7ce515640 1 -- 172.21.15.102:0/2660169464 shutdown_connections 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.574+0000 7ff7ce515640 1 -- 172.21.15.102:0/2660169464 wait complete. 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.575+0000 7ff7ce515640 10 monclient: build_initial_monmap 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.575+0000 7ff7ce515640 10 monclient: monmap: 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:27:54.577 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.576+0000 7ff7ce515640 1 librados: starting msgr at 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.576+0000 7ff7ce515640 1 librados: starting objecter 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.576+0000 7ff7ce515640 1 Processor -- start 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.576+0000 7ff7ce515640 1 -- start start 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.576+0000 7ff7ce515640 1 librados: setting wanted keys 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.576+0000 7ff7ce515640 1 librados: calling monclient init 2024-03-25T18:27:54.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.576+0000 7ff7ce515640 10 monclient: init 2024-03-25T18:27:54.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:54.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:54.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 10 monclient(hunting): picked mon.noname-a con 0x7ff7c80b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:27:54.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 10 monclient(hunting): picked mon.noname-c con 0x7ff7c80b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:27:54.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 10 monclient(hunting): picked mon.noname-b con 0x7ff7c814be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:27:54.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c813e050 con 0x7ff7c80b1730 2024-03-25T18:27:54.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c81452a0 con 0x7ff7c814be90 2024-03-25T18:27:54.581 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c813db70 con 0x7ff7c80b4d60 2024-03-25T18:27:54.581 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:54.581 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.577+0000 7ff7ce515640 10 monclient(hunting): authenticate will time out at 7505.865472s 2024-03-25T18:27:54.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.590+0000 7ff7c7fff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff7c80b1730 0x7ff7c81d78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46142/0 (socket says 172.21.15.102:46142) 2024-03-25T18:27:54.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.590+0000 7ff7c7fff640 1 -- 172.21.15.102:0/2921939995 learned_addr learned my addr 172.21.15.102:0/2921939995 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:27:54.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.594+0000 7ff7c4ff9640 10 client.?.objecter ms_handle_connect 0x7ff7c80b1730 2024-03-25T18:27:54.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.594+0000 7ff7c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:54.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.597+0000 7ff7c4ff9640 10 client.?.objecter ms_handle_connect 0x7ff7c814be90 2024-03-25T18:27:54.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.597+0000 7ff7c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:54.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.605+0000 7ff7c4ff9640 10 client.?.objecter ms_handle_connect 0x7ff7c80b4d60 2024-03-25T18:27:54.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.605+0000 7ff7c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:54.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.608+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173329357 0 0) 0x7ff7c813e050 con 0x7ff7c80b1730 2024-03-25T18:27:54.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.608+0000 7ff7c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.608+0000 7ff7c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.608+0000 7ff7c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.608+0000 7ff7c4ff9640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:27:54.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.608+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b4004eb0 con 0x7ff7c80b1730 2024-03-25T18:27:54.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.611+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198447607 0 0) 0x7ff7c81452a0 con 0x7ff7c814be90 2024-03-25T18:27:54.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.611+0000 7ff7c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.611+0000 7ff7c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.611+0000 7ff7c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.611+0000 7ff7c4ff9640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:27:54.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.611+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c813e050 con 0x7ff7c814be90 2024-03-25T18:27:54.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.624+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678447752 0 0) 0x7ff7c813db70 con 0x7ff7c80b4d60 2024-03-25T18:27:54.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:54.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:54.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:54.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c81452a0 con 0x7ff7c80b4d60 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596444253 0 0) 0x7ff7b4004eb0 con 0x7ff7c80b1730 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7c813db70 con 0x7ff7c80b1730 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff7b8001690 con 0x7ff7c80b1730 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:54.627 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:54.628 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:54.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.625+0000 7ff7c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:54.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.643+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63993944 0 0) 0x7ff7c813e050 con 0x7ff7c814be90 2024-03-25T18:27:54.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.643+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7b4004eb0 con 0x7ff7c814be90 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.643+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff7c0001b90 con 0x7ff7c814be90 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.643+0000 7ff7c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.643+0000 7ff7c4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.643+0000 7ff7c4ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:54.644 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:54.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1294235558 0 0) 0x7ff7c813db70 con 0x7ff7c80b1730 2024-03-25T18:27:54.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 >> v1:172.21.15.102:6790/0 conn(0x7ff7c80b4d60 legacy=0x7ff7c81d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 >> v1:172.21.15.146:6789/0 conn(0x7ff7c814be90 legacy=0x7ff7c81e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:54.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:54.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 1 monclient: found mon.a 2024-03-25T18:27:54.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c813dd00 con 0x7ff7c80b1730 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 5 monclient: authenticate success, global_id 4320 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 10 monclient: _renew_subs 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c81455a0 con 0x7ff7c80b1730 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 10 monclient: _renew_subs 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c8135e80 con 0x7ff7c80b1730 2024-03-25T18:27:54.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.646+0000 7ff7ce515640 1 librados: init done 2024-03-25T18:27:54.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.647+0000 7ff7ce515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:54.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.647+0000 7ff7ce515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:54.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.647+0000 7ff7ce515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:54.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.647+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c80627b0 con 0x7ff7c80b1730 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.946+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7b80039a0 con 0x7ff7c80b1730 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.946+0000 7ff7c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.946+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff7b8001cc0 con 0x7ff7c80b1730 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.946+0000 7ff7c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.946+0000 7ff7c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.946+0000 7ff7c4ff9640 10 monclient: dump: 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.947 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:27:54.948 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:54.948 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:27:54.948 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:27:54.948 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:27:54.948 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:27:54.948 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:27:54.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.946+0000 7ff7c67fc640 4 set_mon_vals no callback set 2024-03-25T18:27:54.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.947+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7ff7b8003eb0 con 0x7ff7c80b1730 2024-03-25T18:27:54.949 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.947+0000 7ff7c77fe640 10 monclient: get_auth_request con 0x7ff7b406df70 auth_method 0 2024-03-25T18:27:54.949 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.948+0000 7ff7c4ff9640 10 client.4320.objecter ms_handle_connect 0x7ff7b406df70 2024-03-25T18:27:54.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.954+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7ff7b8054910 con 0x7ff7c80b1730 2024-03-25T18:27:54.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.954+0000 7ff7c4ff9640 10 client.4320.objecter ms_dispatch 0x7ff7c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:54.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.954+0000 7ff7c4ff9640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:54.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.954+0000 7ff7c4ff9640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:54.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.954+0000 7ff7c4ff9640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-25T18:27:54.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.954+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7b8050420 con 0x7ff7c80b1730 2024-03-25T18:27:54.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.954+0000 7ff7c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:54.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:54.954+0000 7ff7c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:55.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.203+0000 7ff7ce515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:55.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.203+0000 7ff7ce515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:55.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.203+0000 7ff7ce515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:27:55.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.203+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7c8069f40 con 0x7ff7c80b1730 2024-03-25T18:27:55.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.233+0000 7ff7c4ff9640 1 -- 172.21.15.102:0/2921939995 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7ff7b8052350 con 0x7ff7c80b1730 2024-03-25T18:27:55.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.233+0000 7ff7c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:55.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.233+0000 7ff7c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:55.235 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:27:55.235 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:55.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 10 librados: watch_flush enter 2024-03-25T18:27:55.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 10 librados: watch_flush exit 2024-03-25T18:27:55.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-25T18:27:55.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 10 client.4320.objecter successfully canceled tick 2024-03-25T18:27:55.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 >> v1:172.21.15.146:6800/3848252714 conn(0x7ff7b406df70 legacy=0x7ff7b4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 10 monclient: shutdown 2024-03-25T18:27:55.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:55.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 >> v1:172.21.15.102:6789/0 conn(0x7ff7c80b1730 legacy=0x7ff7c81d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 shutdown_connections 2024-03-25T18:27:55.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 >> 172.21.15.102:0/2921939995 conn(0x7ff7c80617a0 msgr2=0x7ff7c813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:55.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 shutdown_connections 2024-03-25T18:27:55.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 1 -- 172.21.15.102:0/2921939995 wait complete. 2024-03-25T18:27:55.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:27:55.236+0000 7ff7ce515640 1 librados: shutdown 2024-03-25T18:27:55.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:58.083 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:14.319 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.138 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:14.374 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.167 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:14.403 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.188 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:14.423 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.196 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:14.410 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.197 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:14.433 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.204 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:14.418 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.229 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:14.440 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.237 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:14.473 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:02.056 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:02.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.172+0000 7ff5e559d640 10 monclient: get_monmap_and_config 2024-03-25T18:28:02.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.172+0000 7ff5e559d640 10 monclient: build_initial_monmap 2024-03-25T18:28:02.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.172+0000 7ff5e559d640 10 monclient: monmap: 2024-03-25T18:28:02.173 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:02.173 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:02.173 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:28:02.173218+0000 2024-03-25T18:28:02.173 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:28:02.173218+0000 2024-03-25T18:28:02.174 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:02.174 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:02.174 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:02.174 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:28:02.174 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:28:02.174 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:02.175 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.173+0000 7ff5e559d640 1 Processor -- start 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.173+0000 7ff5e559d640 1 -- start start 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.173+0000 7ff5e559d640 10 monclient: init 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5e006f380 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5e006f760 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5e0072670 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e00718b0 con 0x7ff5e0072670 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e013efc0 con 0x7ff5e006f760 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e0070120 con 0x7ff5e006f380 2024-03-25T18:28:02.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:02.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.174+0000 7ff5e559d640 10 monclient(hunting): authenticate will time out at 7513.462601s 2024-03-25T18:28:02.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.180+0000 7ff5df7fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff5e0072670 0x7ff5e014a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58500/0 (socket says 172.21.15.102:58500) 2024-03-25T18:28:02.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.180+0000 7ff5df7fe640 1 -- 172.21.15.102:0/2793127461 learned_addr learned my addr 172.21.15.102:0/2793127461 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:02.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.199+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748673010 0 0) 0x7ff5e00718b0 con 0x7ff5e0072670 2024-03-25T18:28:02.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.199+0000 7ff5de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.199+0000 7ff5de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.199+0000 7ff5de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.199+0000 7ff5de7fc640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:28:02.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.199+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5cc003520 con 0x7ff5e0072670 2024-03-25T18:28:02.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.203+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596262991 0 0) 0x7ff5e013efc0 con 0x7ff5e006f760 2024-03-25T18:28:02.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.203+0000 7ff5de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.203+0000 7ff5de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.203+0000 7ff5de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.203+0000 7ff5de7fc640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:28:02.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.203+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e00718b0 con 0x7ff5e006f760 2024-03-25T18:28:02.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 696937541 0 0) 0x7ff5cc003520 con 0x7ff5e0072670 2024-03-25T18:28:02.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 >> v1:172.21.15.102:6790/0 conn(0x7ff5e006f380 legacy=0x7ff5e013f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 >> v1:172.21.15.146:6789/0 conn(0x7ff5e006f760 legacy=0x7ff5e0071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 1 monclient: found mon.noname-a 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e0070610 con 0x7ff5e0072670 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5e559d640 5 monclient: authenticate success, global_id 4323 2024-03-25T18:28:02.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.206+0000 7ff5e559d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.208+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff5d80019d0 con 0x7ff5e0072670 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.208+0000 7ff5de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.208+0000 7ff5de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.208+0000 7ff5de7fc640 10 monclient: dump: 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:02.209 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:02.210 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:02.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.208+0000 7ff5e559d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:02.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.233+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5d8001bb0 con 0x7ff5e0072670 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.233+0000 7ff5de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.233+0000 7ff5de7fc640 1 -- 172.21.15.102:0/2793127461 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff5d8001f40 con 0x7ff5e0072670 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.233+0000 7ff5de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.233+0000 7ff5de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.233+0000 7ff5de7fc640 10 monclient: dump: 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:02.234 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:02.235 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:02.235 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:02.235 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:02.235 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.233+0000 7ff5e559d640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:02.235 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.233+0000 7ff5e559d640 4 set_mon_vals no callback set 2024-03-25T18:28:02.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 10 monclient: shutdown 2024-03-25T18:28:02.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:02.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 1 -- 172.21.15.102:0/2793127461 >> v1:172.21.15.102:6789/0 conn(0x7ff5e0072670 legacy=0x7ff5e014a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 1 -- 172.21.15.102:0/2793127461 shutdown_connections 2024-03-25T18:28:02.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 1 -- 172.21.15.102:0/2793127461 >> 172.21.15.102:0/2793127461 conn(0x7ff5e006c830 msgr2=0x7ff5e00b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 1 -- 172.21.15.102:0/2793127461 shutdown_connections 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 1 -- 172.21.15.102:0/2793127461 wait complete. 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 10 monclient: build_initial_monmap 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.234+0000 7ff5e559d640 10 monclient: monmap: 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:28:02.238 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.235+0000 7ff5e559d640 1 librados: starting msgr at 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.235+0000 7ff5e559d640 1 librados: starting objecter 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.235+0000 7ff5e559d640 1 Processor -- start 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.235+0000 7ff5e559d640 1 -- start start 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.235+0000 7ff5e559d640 1 librados: setting wanted keys 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.235+0000 7ff5e559d640 1 librados: calling monclient init 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.235+0000 7ff5e559d640 10 monclient: init 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 10 monclient(hunting): picked mon.noname-c con 0x7ff5e006f380 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 10 monclient(hunting): picked mon.noname-a con 0x7ff5e006f760 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 10 monclient(hunting): picked mon.noname-b con 0x7ff5e0072670 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:02.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e0138170 con 0x7ff5e006f760 2024-03-25T18:28:02.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e01346f0 con 0x7ff5e0072670 2024-03-25T18:28:02.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e0138600 con 0x7ff5e006f380 2024-03-25T18:28:02.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:02.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.236+0000 7ff5e559d640 10 monclient(hunting): authenticate will time out at 7513.524280s 2024-03-25T18:28:02.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.254+0000 7ff5df7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff5e0072670 0x7ff5e0133fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:59130/0 (socket says 172.21.15.102:59130) 2024-03-25T18:28:02.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.254+0000 7ff5df7fe640 1 -- 172.21.15.102:0/1169620277 learned_addr learned my addr 172.21.15.102:0/1169620277 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:02.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.256+0000 7ff5d77fe640 10 client.?.objecter ms_handle_connect 0x7ff5e0072670 2024-03-25T18:28:02.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.256+0000 7ff5d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:02.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.257+0000 7ff5d77fe640 10 client.?.objecter ms_handle_connect 0x7ff5e006f760 2024-03-25T18:28:02.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.257+0000 7ff5d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:02.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.266+0000 7ff5d77fe640 10 client.?.objecter ms_handle_connect 0x7ff5e006f380 2024-03-25T18:28:02.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.266+0000 7ff5d77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:02.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.266+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066165768 0 0) 0x7ff5e01346f0 con 0x7ff5e0072670 2024-03-25T18:28:02.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.266+0000 7ff5d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.266+0000 7ff5d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.266+0000 7ff5d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.266+0000 7ff5d77fe640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:28:02.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.266+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c0004eb0 con 0x7ff5e0072670 2024-03-25T18:28:02.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.273+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119727810 0 0) 0x7ff5e0138170 con 0x7ff5e006f760 2024-03-25T18:28:02.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.273+0000 7ff5d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.273+0000 7ff5d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.273+0000 7ff5d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.273+0000 7ff5d77fe640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:28:02.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.273+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e01346f0 con 0x7ff5e006f760 2024-03-25T18:28:02.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.275+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1605011144 0 0) 0x7ff5c0004eb0 con 0x7ff5e0072670 2024-03-25T18:28:02.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.275+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5e0138170 con 0x7ff5e0072670 2024-03-25T18:28:02.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.275+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff5d8001f80 con 0x7ff5e0072670 2024-03-25T18:28:02.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.275+0000 7ff5d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.275+0000 7ff5d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:02.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.275+0000 7ff5d77fe640 10 monclient(hunting): dump: 2024-03-25T18:28:02.276 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:02.276 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:02.277 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.275+0000 7ff5d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:02.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.276+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932161653 0 0) 0x7ff5e0138600 con 0x7ff5e006f380 2024-03-25T18:28:02.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.276+0000 7ff5d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:02.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.276+0000 7ff5d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:02.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.276+0000 7ff5d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:02.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.276+0000 7ff5d77fe640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:28:02.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.276+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c0004eb0 con 0x7ff5e006f380 2024-03-25T18:28:02.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.294+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3139453625 0 0) 0x7ff5e01346f0 con 0x7ff5e006f760 2024-03-25T18:28:02.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.294+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5e0138600 con 0x7ff5e006f760 2024-03-25T18:28:02.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.294+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff5c8001930 con 0x7ff5e006f760 2024-03-25T18:28:02.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.294+0000 7ff5d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.294+0000 7ff5d77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:02.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.294+0000 7ff5d77fe640 10 monclient(hunting): dump: 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:02.296 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:02.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.298+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431887541 0 0) 0x7ff5c0004eb0 con 0x7ff5e006f380 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.298+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5e01346f0 con 0x7ff5e006f380 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.298+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff5d0001860 con 0x7ff5e006f380 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.298+0000 7ff5d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.298+0000 7ff5d77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.298+0000 7ff5d77fe640 10 monclient(hunting): dump: 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:02.299 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:02.300 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:02.300 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:02.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.303+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1369237635 0 0) 0x7ff5e0138170 con 0x7ff5e0072670 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.303+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 >> v1:172.21.15.102:6790/0 conn(0x7ff5e006f380 legacy=0x7ff5e01d75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 >> v1:172.21.15.102:6789/0 conn(0x7ff5e006f760 legacy=0x7ff5e01338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 1 monclient: found mon.b 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e0139ce0 con 0x7ff5e0072670 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 10 monclient: _renew_subs 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:02.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e0139fc0 con 0x7ff5e0072670 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 10 monclient: _renew_subs 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e0142ba0 con 0x7ff5e0072670 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.304+0000 7ff5e559d640 1 librados: init done 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.305+0000 7ff5e559d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.305+0000 7ff5e559d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.305+0000 7ff5e559d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:02.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.305+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e0070b00 con 0x7ff5e0072670 2024-03-25T18:28:02.594 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.593+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5d80014f0 con 0x7ff5e0072670 2024-03-25T18:28:02.594 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.593+0000 7ff5d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:02.594 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.593+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff5d8004fb0 con 0x7ff5e0072670 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.593+0000 7ff5d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.594+0000 7ff5d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.594+0000 7ff5d77fe640 10 monclient: dump: 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:02.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.593+0000 7ff5ddffb640 4 set_mon_vals no callback set 2024-03-25T18:28:02.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.594+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7ff5d801afc0 con 0x7ff5e0072670 2024-03-25T18:28:02.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.595+0000 7ff5deffd640 10 monclient: get_auth_request con 0x7ff5c006e160 auth_method 0 2024-03-25T18:28:02.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.596+0000 7ff5d77fe640 10 client.4324.objecter ms_handle_connect 0x7ff5c006e160 2024-03-25T18:28:02.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.600+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7ff5d8054c10 con 0x7ff5e0072670 2024-03-25T18:28:02.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.600+0000 7ff5d77fe640 10 client.4324.objecter ms_dispatch 0x7ff5e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:02.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.600+0000 7ff5d77fe640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:02.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.600+0000 7ff5d77fe640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:02.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.600+0000 7ff5d77fe640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T18:28:02.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.600+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5d8050550 con 0x7ff5e0072670 2024-03-25T18:28:02.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.600+0000 7ff5d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:02.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.600+0000 7ff5d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:02.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.840+0000 7ff5e559d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:02.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.840+0000 7ff5e559d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:02.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.840+0000 7ff5e559d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:02.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.840+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5e0137f00 con 0x7ff5e0072670 2024-03-25T18:28:02.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.875+0000 7ff5d77fe640 1 -- 172.21.15.102:0/1169620277 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7ff5d8052650 con 0x7ff5e0072670 2024-03-25T18:28:02.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.875+0000 7ff5d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:02.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.875+0000 7ff5d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:02.877 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:28:02.877 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 10 librados: watch_flush enter 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 10 librados: watch_flush exit 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 10 client.4324.objecter successfully canceled tick 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 >> v1:172.21.15.146:6800/3848252714 conn(0x7ff5c006e160 legacy=0x7ff5c0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 10 monclient: shutdown 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 >> v1:172.21.15.146:6789/0 conn(0x7ff5e0072670 legacy=0x7ff5e0133fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 shutdown_connections 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 >> 172.21.15.102:0/1169620277 conn(0x7ff5e006c830 msgr2=0x7ff5e00b38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:02.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 shutdown_connections 2024-03-25T18:28:02.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 1 -- 172.21.15.102:0/1169620277 wait complete. 2024-03-25T18:28:02.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:02.878+0000 7ff5e559d640 1 librados: shutdown 2024-03-25T18:28:02.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:09.697 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:09.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.813+0000 7fb6e3cf6640 10 monclient: get_monmap_and_config 2024-03-25T18:28:09.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.813+0000 7fb6e3cf6640 10 monclient: build_initial_monmap 2024-03-25T18:28:09.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.813+0000 7fb6e3cf6640 10 monclient: monmap: 2024-03-25T18:28:09.814 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:09.814 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:09.814 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:28:09.814310+0000 2024-03-25T18:28:09.814 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:28:09.814310+0000 2024-03-25T18:28:09.814 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:09.815 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:09.815 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:09.815 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:28:09.815 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:28:09.815 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:09.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.814+0000 7fb6e3cf6640 1 Processor -- start 2024-03-25T18:28:09.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.814+0000 7fb6e3cf6640 1 -- start start 2024-03-25T18:28:09.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.814+0000 7fb6e3cf6640 10 monclient: init 2024-03-25T18:28:09.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 10 monclient(hunting): picked mon.noname-c con 0x7fb6dc0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 10 monclient(hunting): picked mon.noname-a con 0x7fb6dc13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 10 monclient(hunting): picked mon.noname-b con 0x7fb6dc1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc142520 con 0x7fb6dc13f710 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0b28a0 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0b2a30 con 0x7fb6dc1430c0 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:09.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.815+0000 7fb6e3cf6640 10 monclient(hunting): authenticate will time out at 7521.103643s 2024-03-25T18:28:09.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.824+0000 7fb6e1291640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb6dc13f710 0x7fb6dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35094/0 (socket says 172.21.15.102:35094) 2024-03-25T18:28:09.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.824+0000 7fb6e1291640 1 -- 172.21.15.102:0/4252956790 learned_addr learned my addr 172.21.15.102:0/4252956790 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:09.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.828+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562008549 0 0) 0x7fb6dc142520 con 0x7fb6dc13f710 2024-03-25T18:28:09.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.828+0000 7fb6e0a90640 10 monclient(hunting): _negotiate 2024-03-25T18:28:09.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.828+0000 7fb6e0a90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:09.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.828+0000 7fb6e0a90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:09.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.828+0000 7fb6e0a90640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:28:09.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.828+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c8003520 con 0x7fb6dc13f710 2024-03-25T18:28:09.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.843+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419823928 0 0) 0x7fb6dc0b28a0 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.843+0000 7fb6e0a90640 10 monclient(hunting): _negotiate 2024-03-25T18:28:09.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.843+0000 7fb6e0a90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:09.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.843+0000 7fb6e0a90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:09.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.843+0000 7fb6e0a90640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:28:09.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.843+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc142520 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2767016601 0 0) 0x7fb6c8003520 con 0x7fb6dc13f710 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 >> v1:172.21.15.102:6790/0 conn(0x7fb6dc1430c0 legacy=0x7fb6dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 >> v1:172.21.15.146:6789/0 conn(0x7fb6dc0a98f0 legacy=0x7fb6dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 10 monclient: _finish_hunting 0 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 1 monclient: found mon.noname-a 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0b44c0 con 0x7fb6dc13f710 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 10 monclient: _finish_auth 0 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 10 monclient: _check_auth_tickets 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb6cc001690 con 0x7fb6dc13f710 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:09.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e0a90640 10 monclient: dump: 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e3cf6640 5 monclient: authenticate success, global_id 4329 2024-03-25T18:28:09.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.846+0000 7fb6e3cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.857+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6cc0017f0 con 0x7fb6dc13f710 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.857+0000 7fb6e0a90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.857+0000 7fb6e0a90640 1 -- 172.21.15.102:0/4252956790 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb6cc001c00 con 0x7fb6dc13f710 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.857+0000 7fb6e0a90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.857+0000 7fb6e0a90640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.857+0000 7fb6e0a90640 10 monclient: dump: 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:09.858 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:09.859 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:09.859 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:09.859 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:09.859 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:09.859 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:09.859 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.857+0000 7fb6e3cf6640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:09.859 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.857+0000 7fb6e3cf6640 4 set_mon_vals no callback set 2024-03-25T18:28:09.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.858+0000 7fb6e3cf6640 10 monclient: shutdown 2024-03-25T18:28:09.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.858+0000 7fb6e3cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:09.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.858+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/4252956790 >> v1:172.21.15.102:6789/0 conn(0x7fb6dc13f710 legacy=0x7fb6dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.858+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/4252956790 shutdown_connections 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.858+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/4252956790 >> 172.21.15.102:0/4252956790 conn(0x7fb6dc0b7ea0 msgr2=0x7fb6dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.858+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/4252956790 shutdown_connections 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.858+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/4252956790 wait complete. 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 10 monclient: build_initial_monmap 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 10 monclient: monmap: 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:28:09.862 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 1 librados: starting msgr at 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 1 librados: starting objecter 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 1 Processor -- start 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 1 -- start start 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 1 librados: setting wanted keys 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 1 librados: calling monclient init 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.859+0000 7fb6e3cf6640 10 monclient: init 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 10 monclient(hunting): picked mon.noname-c con 0x7fb6dc0a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 10 monclient(hunting): picked mon.noname-b con 0x7fb6dc13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 10 monclient(hunting): picked mon.noname-a con 0x7fb6dc1430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc1d7d80 con 0x7fb6dc1430c0 2024-03-25T18:28:09.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc1d7f10 con 0x7fb6dc13f710 2024-03-25T18:28:09.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc1d80a0 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:09.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.860+0000 7fb6e3cf6640 10 monclient(hunting): authenticate will time out at 7521.148298s 2024-03-25T18:28:09.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.868+0000 7fb6e1a92640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb6dc0a98f0 0x7fb6dc1d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:49000/0 (socket says 172.21.15.102:49000) 2024-03-25T18:28:09.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.868+0000 7fb6e1a92640 1 -- 172.21.15.102:0/1073353372 learned_addr learned my addr 172.21.15.102:0/1073353372 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:09.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.871+0000 7fb6d27fc640 10 client.?.objecter ms_handle_connect 0x7fb6dc0a98f0 2024-03-25T18:28:09.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.871+0000 7fb6d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:09.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.877+0000 7fb6d27fc640 10 client.?.objecter ms_handle_connect 0x7fb6dc13f710 2024-03-25T18:28:09.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.877+0000 7fb6d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:09.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.879+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837569975 0 0) 0x7fb6dc142520 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.879+0000 7fb6d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:09.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.879+0000 7fb6d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:09.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.879+0000 7fb6d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:09.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.879+0000 7fb6d27fc640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:28:09.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.879+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c0004eb0 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.881 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.881+0000 7fb6d27fc640 10 client.?.objecter ms_handle_connect 0x7fb6dc1430c0 2024-03-25T18:28:09.881 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.881+0000 7fb6d27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:09.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.890+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483235710 0 0) 0x7fb6dc1d7d80 con 0x7fb6dc1430c0 2024-03-25T18:28:09.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.890+0000 7fb6d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:09.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.890+0000 7fb6d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:09.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.890+0000 7fb6d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:09.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.890+0000 7fb6d27fc640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:28:09.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.890+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc142520 con 0x7fb6dc1430c0 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.893+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4156144308 0 0) 0x7fb6c0004eb0 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.893+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6dc1d7d80 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.893+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb6d80018f0 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.893+0000 7fb6d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.893+0000 7fb6d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.893+0000 7fb6d27fc640 10 monclient(hunting): dump: 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:09.894 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:09.895 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:09.895 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:09.895 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:09.895 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:09.895 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:09.895 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:09.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.893+0000 7fb6d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:09.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.899+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253397283 0 0) 0x7fb6dc1d7f10 con 0x7fb6dc13f710 2024-03-25T18:28:09.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.899+0000 7fb6d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:09.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.899+0000 7fb6d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:09.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.899+0000 7fb6d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:09.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.899+0000 7fb6d27fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:28:09.900 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.899+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c0004eb0 con 0x7fb6dc13f710 2024-03-25T18:28:09.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1132784738 0 0) 0x7fb6dc1d7d80 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 >> v1:172.21.15.146:6789/0 conn(0x7fb6dc13f710 legacy=0x7fb6dc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 >> v1:172.21.15.102:6789/0 conn(0x7fb6dc1430c0 legacy=0x7fb6dc135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 1 monclient: found mon.c 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0b7200 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 10 monclient: _renew_subs 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:09.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc0b7450 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.909 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:09.909 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 10 monclient: _renew_subs 2024-03-25T18:28:09.909 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:09.909 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc1e3230 con 0x7fb6dc0a98f0 2024-03-25T18:28:09.909 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.906+0000 7fb6e3cf6640 1 librados: init done 2024-03-25T18:28:09.909 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.908+0000 7fb6e3cf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:09.910 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.908+0000 7fb6e3cf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:09.910 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.908+0000 7fb6e3cf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:09.910 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:09.908+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc1d84a0 con 0x7fb6dc0a98f0 2024-03-25T18:28:10.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.198+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6d8003610 con 0x7fb6dc0a98f0 2024-03-25T18:28:10.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.198+0000 7fb6d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.199+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb6d80039e0 con 0x7fb6dc0a98f0 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.199+0000 7fb6d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.199+0000 7fb6d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.199+0000 7fb6d27fc640 10 monclient: dump: 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:10.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.199+0000 7fb6d3fff640 4 set_mon_vals no callback set 2024-03-25T18:28:10.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.199+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fb6d8003c00 con 0x7fb6dc0a98f0 2024-03-25T18:28:10.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.200+0000 7fb6e1291640 10 monclient: get_auth_request con 0x7fb6c006de00 auth_method 0 2024-03-25T18:28:10.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.200+0000 7fb6d27fc640 10 client.4313.objecter ms_handle_connect 0x7fb6c006de00 2024-03-25T18:28:10.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.205+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fb6d8054760 con 0x7fb6dc0a98f0 2024-03-25T18:28:10.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.205+0000 7fb6d27fc640 10 client.4313.objecter ms_dispatch 0x7fb6dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:10.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.205+0000 7fb6d27fc640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:10.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.205+0000 7fb6d27fc640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:10.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.205+0000 7fb6d27fc640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T18:28:10.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.205+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6d8050270 con 0x7fb6dc0a98f0 2024-03-25T18:28:10.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.205+0000 7fb6d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:10.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.205+0000 7fb6d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:10.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.430+0000 7fb6e3cf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:10.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.430+0000 7fb6e3cf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:10.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.430+0000 7fb6e3cf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:10.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.430+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6dc0627b0 con 0x7fb6dc0a98f0 2024-03-25T18:28:10.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.456+0000 7fb6d27fc640 1 -- 172.21.15.102:0/1073353372 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fb6d80521a0 con 0x7fb6dc0a98f0 2024-03-25T18:28:10.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.456+0000 7fb6d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:10.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.456+0000 7fb6d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:10.457 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:28:10.458 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:10.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 10 librados: watch_flush enter 2024-03-25T18:28:10.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 10 librados: watch_flush exit 2024-03-25T18:28:10.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T18:28:10.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 10 client.4313.objecter successfully canceled tick 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 >> v1:172.21.15.146:6800/3848252714 conn(0x7fb6c006de00 legacy=0x7fb6c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 10 monclient: shutdown 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 >> v1:172.21.15.102:6790/0 conn(0x7fb6dc0a98f0 legacy=0x7fb6dc1d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 shutdown_connections 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 >> 172.21.15.102:0/1073353372 conn(0x7fb6dc0b7ea0 msgr2=0x7fb6dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 shutdown_connections 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 1 -- 172.21.15.102:0/1073353372 wait complete. 2024-03-25T18:28:10.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:10.459+0000 7fb6e3cf6640 1 librados: shutdown 2024-03-25T18:28:10.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:17.281 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:17.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.400+0000 7fdd30332640 10 monclient: get_monmap_and_config 2024-03-25T18:28:17.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.400+0000 7fdd30332640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.400+0000 7fdd30332640 10 monclient: monmap: 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:28:17.401844+0000 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:28:17.401844+0000 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:28:17.402 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:17.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.400+0000 7fdd30332640 1 Processor -- start 2024-03-25T18:28:17.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.400+0000 7fdd30332640 1 -- start start 2024-03-25T18:28:17.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.400+0000 7fdd30332640 10 monclient: init 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 10 monclient(hunting): picked mon.noname-a con 0x7fdd280a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 10 monclient(hunting): picked mon.noname-c con 0x7fdd2813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 10 monclient(hunting): picked mon.noname-b con 0x7fdd281430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd28142520 con 0x7fdd280a98f0 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd280b28a0 con 0x7fdd2813f710 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd280b2a30 con 0x7fdd281430c0 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.401+0000 7fdd30332640 10 monclient(hunting): authenticate will time out at 7528.690692s 2024-03-25T18:28:17.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.403+0000 7fdd2e0ce640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdd280a98f0 0x7fdd280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45442/0 (socket says 172.21.15.102:45442) 2024-03-25T18:28:17.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.403+0000 7fdd2e0ce640 1 -- 172.21.15.102:0/1425438171 learned_addr learned my addr 172.21.15.102:0/1425438171 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:17.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.413+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754798425 0 0) 0x7fdd28142520 con 0x7fdd280a98f0 2024-03-25T18:28:17.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.413+0000 7fdd2d0cc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.413+0000 7fdd2d0cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.413+0000 7fdd2d0cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.413+0000 7fdd2d0cc640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:28:17.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.413+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd14003520 con 0x7fdd280a98f0 2024-03-25T18:28:17.425 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833157136 0 0) 0x7fdd280b2a30 con 0x7fdd281430c0 2024-03-25T18:28:17.425 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.425 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.425 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.425 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd28142520 con 0x7fdd281430c0 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218350114 0 0) 0x7fdd280b28a0 con 0x7fdd2813f710 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd280b2a30 con 0x7fdd2813f710 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2882221767 0 0) 0x7fdd14003520 con 0x7fdd280a98f0 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 >> v1:172.21.15.102:6790/0 conn(0x7fdd281430c0 legacy=0x7fdd2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 >> v1:172.21.15.146:6789/0 conn(0x7fdd2813f710 legacy=0x7fdd281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 monclient: found mon.noname-a 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd280b44c0 con 0x7fdd280a98f0 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdd24001690 con 0x7fdd280a98f0 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd2d0cc640 10 monclient: dump: 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:17.427 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:17.428 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:17.428 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:17.428 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:17.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd30332640 5 monclient: authenticate success, global_id 4335 2024-03-25T18:28:17.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.423+0000 7fdd30332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.430+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd240017f0 con 0x7fdd280a98f0 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.430+0000 7fdd2d0cc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.430+0000 7fdd2d0cc640 1 -- 172.21.15.102:0/1425438171 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdd24001c00 con 0x7fdd280a98f0 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.430+0000 7fdd2d0cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.430+0000 7fdd2d0cc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.430+0000 7fdd2d0cc640 10 monclient: dump: 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:17.432 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:17.433 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:17.433 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:17.433 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:17.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.430+0000 7fdd30332640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:17.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.430+0000 7fdd30332640 4 set_mon_vals no callback set 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.431+0000 7fdd30332640 10 monclient: shutdown 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.431+0000 7fdd30332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.431+0000 7fdd30332640 1 -- 172.21.15.102:0/1425438171 >> v1:172.21.15.102:6789/0 conn(0x7fdd280a98f0 legacy=0x7fdd280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.431+0000 7fdd30332640 1 -- 172.21.15.102:0/1425438171 shutdown_connections 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.431+0000 7fdd30332640 1 -- 172.21.15.102:0/1425438171 >> 172.21.15.102:0/1425438171 conn(0x7fdd280b7ea0 msgr2=0x7fdd280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.431+0000 7fdd30332640 1 -- 172.21.15.102:0/1425438171 shutdown_connections 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.431+0000 7fdd30332640 1 -- 172.21.15.102:0/1425438171 wait complete. 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.432+0000 7fdd30332640 10 monclient: build_initial_monmap 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.432+0000 7fdd30332640 10 monclient: monmap: 2024-03-25T18:28:17.435 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:17.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.432+0000 7fdd30332640 1 librados: starting msgr at 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.432+0000 7fdd30332640 1 librados: starting objecter 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 1 Processor -- start 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 1 -- start start 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 1 librados: setting wanted keys 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 1 librados: calling monclient init 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 10 monclient: init 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 10 monclient(hunting): picked mon.noname-b con 0x7fdd280a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 10 monclient(hunting): picked mon.noname-c con 0x7fdd2813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 10 monclient(hunting): picked mon.noname-a con 0x7fdd281430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd281369b0 con 0x7fdd281430c0 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd281366b0 con 0x7fdd280a98f0 2024-03-25T18:28:17.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd28133ad0 con 0x7fdd2813f710 2024-03-25T18:28:17.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:17.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.433+0000 7fdd30332640 10 monclient(hunting): authenticate will time out at 7528.722652s 2024-03-25T18:28:17.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.442+0000 7fdd2e8cf640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdd281430c0 0x7fdd281333c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45488/0 (socket says 172.21.15.102:45488) 2024-03-25T18:28:17.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.442+0000 7fdd2e8cf640 1 -- 172.21.15.102:0/4056767045 learned_addr learned my addr 172.21.15.102:0/4056767045 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:17.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.443+0000 7fdd1effd640 10 client.?.objecter ms_handle_connect 0x7fdd281430c0 2024-03-25T18:28:17.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.443+0000 7fdd1effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.444+0000 7fdd1effd640 10 client.?.objecter ms_handle_connect 0x7fdd280a98f0 2024-03-25T18:28:17.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.444+0000 7fdd1effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.445+0000 7fdd1effd640 10 client.?.objecter ms_handle_connect 0x7fdd2813f710 2024-03-25T18:28:17.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.445+0000 7fdd1effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:17.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.449+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370017904 0 0) 0x7fdd28133ad0 con 0x7fdd2813f710 2024-03-25T18:28:17.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.449+0000 7fdd1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.449+0000 7fdd1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.449+0000 7fdd1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.449+0000 7fdd1effd640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:28:17.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.450+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd0c004eb0 con 0x7fdd2813f710 2024-03-25T18:28:17.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.455+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778362243 0 0) 0x7fdd281369b0 con 0x7fdd281430c0 2024-03-25T18:28:17.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.455+0000 7fdd1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.455+0000 7fdd1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.455+0000 7fdd1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.455+0000 7fdd1effd640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:28:17.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.455+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd28133ad0 con 0x7fdd281430c0 2024-03-25T18:28:17.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.463+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150586317 0 0) 0x7fdd281366b0 con 0x7fdd280a98f0 2024-03-25T18:28:17.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.463+0000 7fdd1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:17.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.463+0000 7fdd1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:17.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.463+0000 7fdd1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:17.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.463+0000 7fdd1effd640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:28:17.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.463+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd281369b0 con 0x7fdd280a98f0 2024-03-25T18:28:17.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.472+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 702974856 0 0) 0x7fdd0c004eb0 con 0x7fdd2813f710 2024-03-25T18:28:17.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.472+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd281366b0 con 0x7fdd2813f710 2024-03-25T18:28:17.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.472+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdd180018f0 con 0x7fdd2813f710 2024-03-25T18:28:17.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.473+0000 7fdd1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.473+0000 7fdd1effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.473+0000 7fdd1effd640 10 monclient(hunting): dump: 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:17.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.473+0000 7fdd1effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1397725438 0 0) 0x7fdd281369b0 con 0x7fdd280a98f0 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd0c004eb0 con 0x7fdd280a98f0 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdd24001b00 con 0x7fdd280a98f0 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 10 monclient(hunting): dump: 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:17.481 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2749976107 0 0) 0x7fdd28133ad0 con 0x7fdd281430c0 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd281369b0 con 0x7fdd281430c0 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdd20001c10 con 0x7fdd281430c0 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.479+0000 7fdd1effd640 10 monclient(hunting): dump: 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.482 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:17.483 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:17.483 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:17.483 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:17.483 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:17.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2839814627 0 0) 0x7fdd281366b0 con 0x7fdd2813f710 2024-03-25T18:28:17.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 >> v1:172.21.15.146:6789/0 conn(0x7fdd280a98f0 legacy=0x7fdd281371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 >> v1:172.21.15.102:6789/0 conn(0x7fdd281430c0 legacy=0x7fdd281333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:17.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 1 monclient: found mon.c 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28136e40 con 0x7fdd2813f710 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 10 monclient: _finish_auth 0 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 5 monclient: authenticate success, global_id 4319 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 10 monclient: _renew_subs 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd281378f0 con 0x7fdd2813f710 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 10 monclient: _renew_subs 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:17.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd280b1480 con 0x7fdd2813f710 2024-03-25T18:28:17.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.485+0000 7fdd30332640 1 librados: init done 2024-03-25T18:28:17.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.486+0000 7fdd30332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.486+0000 7fdd30332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.486+0000 7fdd30332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:17.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.486+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd28137030 con 0x7fdd2813f710 2024-03-25T18:28:17.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.784+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd18003610 con 0x7fdd2813f710 2024-03-25T18:28:17.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.784+0000 7fdd1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:17.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.784+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdd18003920 con 0x7fdd2813f710 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.784+0000 7fdd1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.784+0000 7fdd1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.784+0000 7fdd1effd640 10 monclient: dump: 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:17.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.784+0000 7fdd2c8cb640 4 set_mon_vals no callback set 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.785+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fdd180198f0 con 0x7fdd2813f710 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.786+0000 7fdd2e0ce640 10 monclient: get_auth_request con 0x7fdd0c06e0b0 auth_method 0 2024-03-25T18:28:17.788 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.786+0000 7fdd1effd640 10 client.4319.objecter ms_handle_connect 0x7fdd0c06e0b0 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.791+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fdd180535c0 con 0x7fdd2813f710 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.791+0000 7fdd1effd640 10 client.4319.objecter ms_dispatch 0x7fdd28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.791+0000 7fdd1effd640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.791+0000 7fdd1effd640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.791+0000 7fdd1effd640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.791+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd1804f0d0 con 0x7fdd2813f710 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.791+0000 7fdd1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:17.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:17.791+0000 7fdd1effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:18.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.044+0000 7fdd30332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.044+0000 7fdd30332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.044+0000 7fdd30332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:18.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.044+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd28136b60 con 0x7fdd2813f710 2024-03-25T18:28:18.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.066+0000 7fdd1effd640 1 -- 172.21.15.102:0/4056767045 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fdd28136b60 con 0x7fdd2813f710 2024-03-25T18:28:18.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.066+0000 7fdd1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:18.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.066+0000 7fdd1effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:18.068 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:28:18.068 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:18.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.068+0000 7fdd30332640 10 librados: watch_flush enter 2024-03-25T18:28:18.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.068+0000 7fdd30332640 10 librados: watch_flush exit 2024-03-25T18:28:18.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.068+0000 7fdd30332640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-25T18:28:18.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.068+0000 7fdd30332640 10 client.4319.objecter successfully canceled tick 2024-03-25T18:28:18.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.068+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 >> v1:172.21.15.146:6800/3848252714 conn(0x7fdd0c06e0b0 legacy=0x7fdd0c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.068+0000 7fdd30332640 10 monclient: shutdown 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.068+0000 7fdd30332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.068+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 >> v1:172.21.15.102:6790/0 conn(0x7fdd2813f710 legacy=0x7fdd28132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.069+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 shutdown_connections 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.069+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 >> 172.21.15.102:0/4056767045 conn(0x7fdd280b7ea0 msgr2=0x7fdd28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.069+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 shutdown_connections 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.069+0000 7fdd30332640 1 -- 172.21.15.102:0/4056767045 wait complete. 2024-03-25T18:28:18.072 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:18.069+0000 7fdd30332640 1 librados: shutdown 2024-03-25T18:28:18.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:18.185 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:34.421 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.186 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:34.422 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.191 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:34.427 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.191 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:34.427 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.236 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:34.472 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.237 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:34.472 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.245 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:34.458 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.249 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:34.463 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.272 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:34.485 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.889 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.001+0000 7f365264e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.001+0000 7f365264e640 10 monclient: build_initial_monmap 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.001+0000 7f365264e640 10 monclient: monmap: 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:28:25.003131+0000 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:28:25.003131+0000 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:25.003 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:28:25.004 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:28:25.004 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.002+0000 7f365264e640 1 Processor -- start 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.002+0000 7f365264e640 1 -- start start 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.002+0000 7f365264e640 10 monclient: init 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.002+0000 7f365264e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.002+0000 7f365264e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.002+0000 7f365264e640 10 monclient(hunting): picked mon.noname-a con 0x7f364c0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.002+0000 7f365264e640 10 monclient(hunting): picked mon.noname-b con 0x7f364c13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.003+0000 7f365264e640 10 monclient(hunting): picked mon.noname-c con 0x7f364c1430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.003+0000 7f365264e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c142520 con 0x7f364c0a98f0 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.003+0000 7f365264e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0b28a0 con 0x7f364c1430c0 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.003+0000 7f365264e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0b2a30 con 0x7f364c13f710 2024-03-25T18:28:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.003+0000 7f365264e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:25.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.003+0000 7f365264e640 10 monclient(hunting): authenticate will time out at 7536.291983s 2024-03-25T18:28:25.007 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.005+0000 7f3650beb640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f364c1430c0 0x7f364c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:40270/0 (socket says 172.21.15.102:40270) 2024-03-25T18:28:25.007 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.005+0000 7f3650beb640 1 -- 172.21.15.102:0/847656034 learned_addr learned my addr 172.21.15.102:0/847656034 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:25.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.021+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494695532 0 0) 0x7f364c0b28a0 con 0x7f364c1430c0 2024-03-25T18:28:25.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.021+0000 7f364affd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.021+0000 7f364affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.021+0000 7f364affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.021+0000 7f364affd640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:28:25.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.021+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3634003520 con 0x7f364c1430c0 2024-03-25T18:28:25.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 734669307 0 0) 0x7f3634003520 con 0x7f364c1430c0 2024-03-25T18:28:25.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 >> v1:172.21.15.102:6790/0 conn(0x7f364c13f710 legacy=0x7f364c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 >> v1:172.21.15.102:6789/0 conn(0x7f364c0a98f0 legacy=0x7f364c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 1 monclient: found mon.noname-c 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c0b44c0 con 0x7f364c1430c0 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 10 monclient: _finish_auth 0 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3644001a10 con 0x7f364c1430c0 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f364affd640 10 monclient: dump: 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.031 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f365264e640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.028+0000 7f365264e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.031+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3644001b70 con 0x7f364c1430c0 2024-03-25T18:28:25.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.031+0000 7f364affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.031+0000 7f364affd640 1 -- 172.21.15.102:0/847656034 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3644001f80 con 0x7f364c1430c0 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.031+0000 7f365264e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.031+0000 7f365264e640 4 set_mon_vals no callback set 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.031+0000 7f364affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.031+0000 7f364affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.031+0000 7f364affd640 10 monclient: dump: 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:25.033 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:25.034 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.032+0000 7f365264e640 10 monclient: shutdown 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.032+0000 7f365264e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.032+0000 7f365264e640 1 -- 172.21.15.102:0/847656034 >> v1:172.21.15.146:6789/0 conn(0x7f364c1430c0 legacy=0x7f364c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.032+0000 7f365264e640 1 -- 172.21.15.102:0/847656034 shutdown_connections 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.032+0000 7f365264e640 1 -- 172.21.15.102:0/847656034 >> 172.21.15.102:0/847656034 conn(0x7f364c0b7ea0 msgr2=0x7f364c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.032+0000 7f365264e640 1 -- 172.21.15.102:0/847656034 shutdown_connections 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.032+0000 7f365264e640 1 -- 172.21.15.102:0/847656034 wait complete. 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 10 monclient: build_initial_monmap 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 10 monclient: monmap: 2024-03-25T18:28:25.035 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:25.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 1 librados: starting msgr at 2024-03-25T18:28:25.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 1 librados: starting objecter 2024-03-25T18:28:25.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 1 Processor -- start 2024-03-25T18:28:25.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 1 -- start start 2024-03-25T18:28:25.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 1 librados: setting wanted keys 2024-03-25T18:28:25.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 1 librados: calling monclient init 2024-03-25T18:28:25.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.033+0000 7f365264e640 10 monclient: init 2024-03-25T18:28:25.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:25.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:25.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 10 monclient(hunting): picked mon.noname-b con 0x7f364c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:25.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 10 monclient(hunting): picked mon.noname-a con 0x7f364c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:25.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 10 monclient(hunting): picked mon.noname-c con 0x7f364c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:25.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c1396e0 con 0x7f364c13f710 2024-03-25T18:28:25.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c138530 con 0x7f364c0a98f0 2024-03-25T18:28:25.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c1399c0 con 0x7f364c1430c0 2024-03-25T18:28:25.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:25.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.034+0000 7f365264e640 10 monclient(hunting): authenticate will time out at 7536.323448s 2024-03-25T18:28:25.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.047+0000 7f3650beb640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f364c1430c0 0x7f364c1e1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:44776/0 (socket says 172.21.15.102:44776) 2024-03-25T18:28:25.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.047+0000 7f3650beb640 1 -- 172.21.15.102:0/220459666 learned_addr learned my addr 172.21.15.102:0/220459666 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:25.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.051+0000 7f3648ff9640 10 client.?.objecter ms_handle_connect 0x7f364c1430c0 2024-03-25T18:28:25.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.051+0000 7f3648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:25.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.054+0000 7f3648ff9640 10 client.?.objecter ms_handle_connect 0x7f364c0a98f0 2024-03-25T18:28:25.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.054+0000 7f3648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:25.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.055+0000 7f3648ff9640 10 client.?.objecter ms_handle_connect 0x7f364c13f710 2024-03-25T18:28:25.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.055+0000 7f3648ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:25.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.059+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001585714 0 0) 0x7f364c138530 con 0x7f364c0a98f0 2024-03-25T18:28:25.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.059+0000 7f3648ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.059+0000 7f3648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.059+0000 7f3648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.059+0000 7f3648ff9640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:28:25.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.059+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3638004eb0 con 0x7f364c0a98f0 2024-03-25T18:28:25.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.068+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460337446 0 0) 0x7f364c1399c0 con 0x7f364c1430c0 2024-03-25T18:28:25.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.068+0000 7f3648ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.068+0000 7f3648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.068+0000 7f3648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.068+0000 7f3648ff9640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:28:25.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.068+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c138530 con 0x7f364c1430c0 2024-03-25T18:28:25.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.069+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840978350 0 0) 0x7f364c1396e0 con 0x7f364c13f710 2024-03-25T18:28:25.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.069+0000 7f3648ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:25.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.069+0000 7f3648ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:25.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.069+0000 7f3648ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:25.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.069+0000 7f3648ff9640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:28:25.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.069+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c1399c0 con 0x7f364c13f710 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.076+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 636535035 0 0) 0x7f3638004eb0 con 0x7f364c0a98f0 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.076+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f364c1396e0 con 0x7f364c0a98f0 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.076+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f363c0018c0 con 0x7f364c0a98f0 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.076+0000 7f3648ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.076+0000 7f3648ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.076+0000 7f3648ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.078 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.079 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:25.079 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:25.079 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:25.079 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:25.079 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:25.079 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.076+0000 7f3648ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.079+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2776974706 0 0) 0x7f364c1399c0 con 0x7f364c13f710 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.079+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3638004eb0 con 0x7f364c13f710 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.079+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3640001830 con 0x7f364c13f710 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.079+0000 7f3648ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.079+0000 7f3648ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.079+0000 7f3648ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.081 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.082 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.082 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:25.082 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:25.082 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:25.082 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:25.082 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:25.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.087+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1699218815 0 0) 0x7f364c138530 con 0x7f364c1430c0 2024-03-25T18:28:25.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.087+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f364c1399c0 con 0x7f364c1430c0 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.087+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3644001ec0 con 0x7f364c1430c0 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.087+0000 7f3648ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.087+0000 7f3648ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.087+0000 7f3648ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:25.089 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:25.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 877793313 0 0) 0x7f364c1396e0 con 0x7f364c0a98f0 2024-03-25T18:28:25.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 >> v1:172.21.15.102:6790/0 conn(0x7f364c1430c0 legacy=0x7f364c1e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 >> v1:172.21.15.102:6789/0 conn(0x7f364c13f710 legacy=0x7f364c1da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:25.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 1 monclient: found mon.b 2024-03-25T18:28:25.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:25.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:25.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c138a50 con 0x7f364c0a98f0 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f3648ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 10 monclient: _renew_subs 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c1380a0 con 0x7f364c0a98f0 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 10 monclient: _renew_subs 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c1db140 con 0x7f364c0a98f0 2024-03-25T18:28:25.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.093+0000 7f365264e640 1 librados: init done 2024-03-25T18:28:25.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.094+0000 7f365264e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:25.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.094+0000 7f365264e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:25.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.094+0000 7f365264e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:25.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.094+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c138ca0 con 0x7f364c0a98f0 2024-03-25T18:28:25.361 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.360+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f363c0035f0 con 0x7f364c0a98f0 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.360+0000 7f3648ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.360+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f363c003760 con 0x7f364c0a98f0 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.360+0000 7f3648ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.360+0000 7f3648ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.360+0000 7f3648ff9640 10 monclient: dump: 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:25.362 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:25.363 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:25.363 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:25.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.360+0000 7f364a7fc640 4 set_mon_vals no callback set 2024-03-25T18:28:25.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.360+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f363c019730 con 0x7f364c0a98f0 2024-03-25T18:28:25.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.361+0000 7f364b7fe640 10 monclient: get_auth_request con 0x7f363806e0b0 auth_method 0 2024-03-25T18:28:25.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.361+0000 7f3648ff9640 10 client.4342.objecter ms_handle_connect 0x7f363806e0b0 2024-03-25T18:28:25.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.365+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f363c053410 con 0x7f364c0a98f0 2024-03-25T18:28:25.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.365+0000 7f3648ff9640 10 client.4342.objecter ms_dispatch 0x7f364c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:25.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.365+0000 7f3648ff9640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:25.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.365+0000 7f3648ff9640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:25.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.365+0000 7f3648ff9640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T18:28:25.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.365+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f363c04ed50 con 0x7f364c0a98f0 2024-03-25T18:28:25.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.365+0000 7f3648ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:25.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.365+0000 7f3648ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:25.594 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.592+0000 7f365264e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:25.594 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.592+0000 7f365264e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:25.594 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.592+0000 7f365264e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:25.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.592+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f364c1386c0 con 0x7f364c0a98f0 2024-03-25T18:28:25.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.620+0000 7f3648ff9640 1 -- 172.21.15.102:0/220459666 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f363c050e50 con 0x7f364c0a98f0 2024-03-25T18:28:25.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.620+0000 7f3648ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:25.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.620+0000 7f3648ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:25.623 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:28:25.623 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:25.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 10 librados: watch_flush enter 2024-03-25T18:28:25.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 10 librados: watch_flush exit 2024-03-25T18:28:25.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T18:28:25.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 10 client.4342.objecter successfully canceled tick 2024-03-25T18:28:25.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 >> v1:172.21.15.146:6800/3848252714 conn(0x7f363806e0b0 legacy=0x7f3638070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 10 monclient: shutdown 2024-03-25T18:28:25.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:25.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 >> v1:172.21.15.146:6789/0 conn(0x7f364c0a98f0 legacy=0x7f364c137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:25.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 shutdown_connections 2024-03-25T18:28:25.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 >> 172.21.15.102:0/220459666 conn(0x7f364c0b7ea0 msgr2=0x7f364c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:25.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 shutdown_connections 2024-03-25T18:28:25.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 1 -- 172.21.15.102:0/220459666 wait complete. 2024-03-25T18:28:25.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:25.623+0000 7f365264e640 1 librados: shutdown 2024-03-25T18:28:25.637 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:32.444 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.556+0000 7f183ada3640 10 monclient: get_monmap_and_config 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.556+0000 7f183ada3640 10 monclient: build_initial_monmap 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.556+0000 7f183ada3640 10 monclient: monmap: 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:28:32.557248+0000 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:28:32.557248+0000 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:32.557 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:32.558 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:32.558 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:28:32.558 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:28:32.558 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.557+0000 7f183ada3640 1 Processor -- start 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.557+0000 7f183ada3640 1 -- start start 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.557+0000 7f183ada3640 10 monclient: init 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 10 monclient(hunting): picked mon.noname-a con 0x7f18340a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 10 monclient(hunting): picked mon.noname-c con 0x7f183413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 10 monclient(hunting): picked mon.noname-b con 0x7f18341430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1834142520 con 0x7f18340a98f0 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18340b28a0 con 0x7f183413f710 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18340b2a30 con 0x7f18341430c0 2024-03-25T18:28:32.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:32.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.558+0000 7f183ada3640 10 monclient(hunting): authenticate will time out at 7543.846305s 2024-03-25T18:28:32.566 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.566+0000 7f1833fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f183413f710 0x7f18341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:33668/0 (socket says 172.21.15.102:33668) 2024-03-25T18:28:32.566 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.566+0000 7f1833fff640 1 -- 172.21.15.102:0/4045214749 learned_addr learned my addr 172.21.15.102:0/4045214749 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:32.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.575+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778315424 0 0) 0x7f18340b28a0 con 0x7f183413f710 2024-03-25T18:28:32.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.576+0000 7f18337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:32.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.576+0000 7f18337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:32.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.576+0000 7f18337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:32.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.576+0000 7f18337fe640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:28:32.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.576+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c003520 con 0x7f183413f710 2024-03-25T18:28:32.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.583+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451709356 0 0) 0x7f1834142520 con 0x7f18340a98f0 2024-03-25T18:28:32.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.583+0000 7f18337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:32.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.583+0000 7f18337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:32.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.583+0000 7f18337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:32.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.583+0000 7f18337fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:28:32.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.583+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18340b28a0 con 0x7f18340a98f0 2024-03-25T18:28:32.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.586+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892413676 0 0) 0x7f18340b2a30 con 0x7f18341430c0 2024-03-25T18:28:32.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.586+0000 7f18337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:32.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.586+0000 7f18337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:32.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.586+0000 7f18337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:32.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.586+0000 7f18337fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:28:32.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.586+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1834142520 con 0x7f18341430c0 2024-03-25T18:28:32.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3239816455 0 0) 0x7f181c003520 con 0x7f183413f710 2024-03-25T18:28:32.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 >> v1:172.21.15.102:6790/0 conn(0x7f18341430c0 legacy=0x7f183414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 >> v1:172.21.15.102:6789/0 conn(0x7f18340a98f0 legacy=0x7f18340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:32.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 1 monclient: found mon.noname-c 2024-03-25T18:28:32.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:32.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:28:32.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18340b44c0 con 0x7f183413f710 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1828001690 con 0x7f183413f710 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f183ada3640 5 monclient: authenticate success, global_id 4345 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f183ada3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f18337fe640 10 monclient: dump: 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:32.591 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:32.592 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:32.592 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:32.592 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:32.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.589+0000 7f183ada3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:32.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18280017f0 con 0x7f183413f710 2024-03-25T18:28:32.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f18337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:32.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f18337fe640 1 -- 172.21.15.102:0/4045214749 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1828001c00 con 0x7f183413f710 2024-03-25T18:28:32.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f183ada3640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f183ada3640 4 set_mon_vals no callback set 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f18337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f18337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f18337fe640 10 monclient: dump: 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:32.601 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f183ada3640 10 monclient: shutdown 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f183ada3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.600+0000 7f183ada3640 1 -- 172.21.15.102:0/4045214749 >> v1:172.21.15.146:6789/0 conn(0x7f183413f710 legacy=0x7f18341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.601+0000 7f183ada3640 1 -- 172.21.15.102:0/4045214749 shutdown_connections 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.601+0000 7f183ada3640 1 -- 172.21.15.102:0/4045214749 >> 172.21.15.102:0/4045214749 conn(0x7f18340b7ea0 msgr2=0x7f18340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.601+0000 7f183ada3640 1 -- 172.21.15.102:0/4045214749 shutdown_connections 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.601+0000 7f183ada3640 1 -- 172.21.15.102:0/4045214749 wait complete. 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.601+0000 7f183ada3640 10 monclient: build_initial_monmap 2024-03-25T18:28:32.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.601+0000 7f183ada3640 10 monclient: monmap: 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.602+0000 7f183ada3640 1 librados: starting msgr at 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.602+0000 7f183ada3640 1 librados: starting objecter 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.602+0000 7f183ada3640 1 Processor -- start 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.602+0000 7f183ada3640 1 -- start start 2024-03-25T18:28:32.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.602+0000 7f183ada3640 1 librados: setting wanted keys 2024-03-25T18:28:32.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.602+0000 7f183ada3640 1 librados: calling monclient init 2024-03-25T18:28:32.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.602+0000 7f183ada3640 10 monclient: init 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 10 monclient(hunting): picked mon.noname-a con 0x7f18340a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 10 monclient(hunting): picked mon.noname-b con 0x7f183413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 10 monclient(hunting): picked mon.noname-c con 0x7f18341430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18341d7750 con 0x7f18340a98f0 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18341d85d0 con 0x7f183413f710 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18341d79a0 con 0x7f18341430c0 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:32.606 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.603+0000 7f183ada3640 10 monclient(hunting): authenticate will time out at 7543.891533s 2024-03-25T18:28:32.607 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.607+0000 7f1833fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f183413f710 0x7f18341d4990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:33694/0 (socket says 172.21.15.102:33694) 2024-03-25T18:28:32.607 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.607+0000 7f1833fff640 1 -- 172.21.15.102:0/976217980 learned_addr learned my addr 172.21.15.102:0/976217980 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:32.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.608+0000 7f18317fa640 10 client.?.objecter ms_handle_connect 0x7f183413f710 2024-03-25T18:28:32.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.608+0000 7f18317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:32.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.612+0000 7f18317fa640 10 client.?.objecter ms_handle_connect 0x7f18341430c0 2024-03-25T18:28:32.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.612+0000 7f18317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:32.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.620+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853353389 0 0) 0x7f18341d85d0 con 0x7f183413f710 2024-03-25T18:28:32.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.620+0000 7f18317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:32.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.620+0000 7f18317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:32.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.620+0000 7f18317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:32.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.620+0000 7f18317fa640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:28:32.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.620+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1820004eb0 con 0x7f183413f710 2024-03-25T18:28:32.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.621+0000 7f18317fa640 10 client.?.objecter ms_handle_connect 0x7f18340a98f0 2024-03-25T18:28:32.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.621+0000 7f18317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:32.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.625+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331234743 0 0) 0x7f18341d79a0 con 0x7f18341430c0 2024-03-25T18:28:32.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.625+0000 7f18317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:32.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.625+0000 7f18317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:32.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.625+0000 7f18317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:32.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.625+0000 7f18317fa640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:28:32.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.625+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18200051d0 con 0x7f18341430c0 2024-03-25T18:28:32.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.629+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797885665 0 0) 0x7f18341d7750 con 0x7f18340a98f0 2024-03-25T18:28:32.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.629+0000 7f18317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:32.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.629+0000 7f18317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:32.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.629+0000 7f18317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:32.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.629+0000 7f18317fa640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:28:32.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.629+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18341d79a0 con 0x7f18340a98f0 2024-03-25T18:28:32.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.643+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2509529793 0 0) 0x7f1820004eb0 con 0x7f183413f710 2024-03-25T18:28:32.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.643+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18341d7750 con 0x7f183413f710 2024-03-25T18:28:32.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.643+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1828001b40 con 0x7f183413f710 2024-03-25T18:28:32.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.643+0000 7f18317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:32.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.643+0000 7f18317fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.643+0000 7f18317fa640 10 monclient(hunting): dump: 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.643+0000 7f18317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.645+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2357388144 0 0) 0x7f18341d79a0 con 0x7f18340a98f0 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.645+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1820004eb0 con 0x7f18340a98f0 2024-03-25T18:28:32.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.645+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1824001cb0 con 0x7f18340a98f0 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.645+0000 7f18317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.645+0000 7f18317fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.645+0000 7f18317fa640 10 monclient(hunting): dump: 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:32.648 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:32.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1934628488 0 0) 0x7f1820004eb0 con 0x7f18340a98f0 2024-03-25T18:28:32.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 >> v1:172.21.15.102:6790/0 conn(0x7f18341430c0 legacy=0x7f1834138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 >> v1:172.21.15.146:6789/0 conn(0x7f183413f710 legacy=0x7f18341d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:32.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:32.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 1 monclient: found mon.a 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1834137d60 con 0x7f18340a98f0 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f18317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 10 monclient: _renew_subs 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18341d50a0 con 0x7f18340a98f0 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 10 monclient: _renew_subs 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18341d43c0 con 0x7f18340a98f0 2024-03-25T18:28:32.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.656+0000 7f183ada3640 1 librados: init done 2024-03-25T18:28:32.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.657+0000 7f183ada3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:32.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.657+0000 7f183ada3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:32.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.657+0000 7f183ada3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:32.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.657+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18341d7b30 con 0x7f18340a98f0 2024-03-25T18:28:32.966 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.965+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1824003990 con 0x7f18340a98f0 2024-03-25T18:28:32.966 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.965+0000 7f18317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:32.966 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.965+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1824003ca0 con 0x7f18340a98f0 2024-03-25T18:28:32.966 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.965+0000 7f18317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:32.966 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.965+0000 7f18317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:32.966 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.965+0000 7f18317fa640 10 monclient: dump: 2024-03-25T18:28:32.966 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:32.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.965+0000 7f1832ffd640 4 set_mon_vals no callback set 2024-03-25T18:28:32.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.966+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f1824019c70 con 0x7f18340a98f0 2024-03-25T18:28:32.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.967+0000 7f1833fff640 10 monclient: get_auth_request con 0x7f182006e0c0 auth_method 0 2024-03-25T18:28:32.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.967+0000 7f18317fa640 10 client.4350.objecter ms_handle_connect 0x7f182006e0c0 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.973+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f1824053920 con 0x7f18340a98f0 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.973+0000 7f18317fa640 10 client.4350.objecter ms_dispatch 0x7f1834000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.974+0000 7f18317fa640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.974+0000 7f18317fa640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.974+0000 7f18317fa640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:28:32.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.974+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f182404f260 con 0x7f18340a98f0 2024-03-25T18:28:32.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.974+0000 7f18317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:32.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:32.974+0000 7f18317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:33.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.226+0000 7f183ada3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:33.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.226+0000 7f183ada3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:33.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.226+0000 7f183ada3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:33.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.226+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1834136b20 con 0x7f18340a98f0 2024-03-25T18:28:33.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.334+0000 7f18317fa640 1 -- 172.21.15.102:0/976217980 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f1824051360 con 0x7f18340a98f0 2024-03-25T18:28:33.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.334+0000 7f18317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:33.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.334+0000 7f18317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:33.335 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:28:33.335 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:33.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 10 librados: watch_flush enter 2024-03-25T18:28:33.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 10 librados: watch_flush exit 2024-03-25T18:28:33.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:28:33.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:28:33.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 >> v1:172.21.15.146:6800/3848252714 conn(0x7f182006e0c0 legacy=0x7f1820070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 10 monclient: shutdown 2024-03-25T18:28:33.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:33.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 >> v1:172.21.15.102:6789/0 conn(0x7f18340a98f0 legacy=0x7f18341d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 shutdown_connections 2024-03-25T18:28:33.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 >> 172.21.15.102:0/976217980 conn(0x7f18340b7ea0 msgr2=0x7f1834074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:33.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 shutdown_connections 2024-03-25T18:28:33.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.337+0000 7f183ada3640 1 -- 172.21.15.102:0/976217980 wait complete. 2024-03-25T18:28:33.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:33.338+0000 7f183ada3640 1 librados: shutdown 2024-03-25T18:28:33.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:38.235 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:54.471 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.283 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:54.518 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.324 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:02:54.560 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.352 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:54.588 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.357 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:54.594 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.360 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:54.573 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.367 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:54.581 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.398 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:02:54.634 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.407 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:02:54.620 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:40.154 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:40.271 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.271+0000 7f4061b0e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.271+0000 7f4061b0e640 10 monclient: build_initial_monmap 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.271+0000 7f4061b0e640 10 monclient: monmap: 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:28:40.271776+0000 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:28:40.271776+0000 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:28:40.272 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:40.273 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 1 Processor -- start 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 1 -- start start 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 10 monclient: init 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f405c0a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f405c13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f405c1430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c142520 con 0x7f405c1430c0 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0b28a0 con 0x7f405c13f710 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0b2a30 con 0x7f405c0a98f0 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:40.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.272+0000 7f4061b0e640 10 monclient(hunting): authenticate will time out at 7551.560615s 2024-03-25T18:28:40.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.287+0000 7f405bfff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f405c1430c0 0x7f405c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57556/0 (socket says 172.21.15.102:57556) 2024-03-25T18:28:40.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.287+0000 7f405bfff640 1 -- 172.21.15.102:0/136587760 learned_addr learned my addr 172.21.15.102:0/136587760 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:40.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.305+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139743461 0 0) 0x7f405c142520 con 0x7f405c1430c0 2024-03-25T18:28:40.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.305+0000 7f405a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.305+0000 7f405a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.305+0000 7f405a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.305+0000 7f405a7fc640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:28:40.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.305+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4044003520 con 0x7f405c1430c0 2024-03-25T18:28:40.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.362+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635467503 0 0) 0x7f405c0b2a30 con 0x7f405c0a98f0 2024-03-25T18:28:40.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.362+0000 7f405a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.362+0000 7f405a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.362+0000 7f405a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.362+0000 7f405a7fc640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:28:40.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.362+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c142520 con 0x7f405c0a98f0 2024-03-25T18:28:40.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 792154938 0 0) 0x7f405c142520 con 0x7f405c0a98f0 2024-03-25T18:28:40.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 >> v1:172.21.15.146:6789/0 conn(0x7f405c13f710 legacy=0x7f405c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 >> v1:172.21.15.102:6789/0 conn(0x7f405c1430c0 legacy=0x7f405c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c0b44c0 con 0x7f405c0a98f0 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f404c001670 con 0x7f405c0a98f0 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f405a7fc640 10 monclient: dump: 2024-03-25T18:28:40.404 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f4061b0e640 5 monclient: authenticate success, global_id 4334 2024-03-25T18:28:40.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.402+0000 7f4061b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.415+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f404c0017d0 con 0x7f405c0a98f0 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.415+0000 7f405a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.415+0000 7f405a7fc640 1 -- 172.21.15.102:0/136587760 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f404c001ba0 con 0x7f405c0a98f0 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.415+0000 7f4061b0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.415+0000 7f4061b0e640 4 set_mon_vals no callback set 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.415+0000 7f405a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.415+0000 7f405a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.415+0000 7f405a7fc640 10 monclient: dump: 2024-03-25T18:28:40.416 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:40.417 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:40.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.416+0000 7f4061b0e640 10 monclient: shutdown 2024-03-25T18:28:40.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.416+0000 7f4061b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:40.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.416+0000 7f4061b0e640 1 -- 172.21.15.102:0/136587760 >> v1:172.21.15.102:6790/0 conn(0x7f405c0a98f0 legacy=0x7f405c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.416+0000 7f4061b0e640 1 -- 172.21.15.102:0/136587760 shutdown_connections 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.416+0000 7f4061b0e640 1 -- 172.21.15.102:0/136587760 >> 172.21.15.102:0/136587760 conn(0x7f405c0b7ea0 msgr2=0x7f405c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.416+0000 7f4061b0e640 1 -- 172.21.15.102:0/136587760 shutdown_connections 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.416+0000 7f4061b0e640 1 -- 172.21.15.102:0/136587760 wait complete. 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 10 monclient: build_initial_monmap 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 10 monclient: monmap: 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:28:40.420 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 1 librados: starting msgr at 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 1 librados: starting objecter 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 1 Processor -- start 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 1 -- start start 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 1 librados: setting wanted keys 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 1 librados: calling monclient init 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.417+0000 7f4061b0e640 10 monclient: init 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:40.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f405c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:40.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f405c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:40.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f405c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:40.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c14e910 con 0x7f405c13f710 2024-03-25T18:28:40.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0abb90 con 0x7f405c0a98f0 2024-03-25T18:28:40.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c14e5c0 con 0x7f405c1430c0 2024-03-25T18:28:40.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:40.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.418+0000 7f4061b0e640 10 monclient(hunting): authenticate will time out at 7551.706198s 2024-03-25T18:28:40.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.426+0000 7f405affd640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f405c13f710 0x7f405c1d7870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57626/0 (socket says 172.21.15.102:57626) 2024-03-25T18:28:40.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.426+0000 7f405affd640 1 -- 172.21.15.102:0/4061937904 learned_addr learned my addr 172.21.15.102:0/4061937904 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:40.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.428+0000 7f4043fff640 10 client.?.objecter ms_handle_connect 0x7f405c13f710 2024-03-25T18:28:40.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.428+0000 7f4043fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:40.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.438+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821869262 0 0) 0x7f405c14e910 con 0x7f405c13f710 2024-03-25T18:28:40.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.438+0000 7f4043fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.438+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.438+0000 7f4043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.438+0000 7f4043fff640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:28:40.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.438+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4048004eb0 con 0x7f405c13f710 2024-03-25T18:28:40.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.439+0000 7f4043fff640 10 client.?.objecter ms_handle_connect 0x7f405c1430c0 2024-03-25T18:28:40.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.439+0000 7f4043fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:40.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.442+0000 7f4043fff640 10 client.?.objecter ms_handle_connect 0x7f405c0a98f0 2024-03-25T18:28:40.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.442+0000 7f4043fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:40.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.451+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275410475 0 0) 0x7f405c14e5c0 con 0x7f405c1430c0 2024-03-25T18:28:40.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.451+0000 7f4043fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.451+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.451+0000 7f4043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.451+0000 7f4043fff640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:28:40.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.451+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c14e910 con 0x7f405c1430c0 2024-03-25T18:28:40.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.453+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710731522 0 0) 0x7f405c0abb90 con 0x7f405c0a98f0 2024-03-25T18:28:40.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.453+0000 7f4043fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:40.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.453+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:40.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.453+0000 7f4043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:40.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.453+0000 7f4043fff640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:28:40.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.453+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c14e5c0 con 0x7f405c0a98f0 2024-03-25T18:28:40.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.456+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007137924 0 0) 0x7f4048004eb0 con 0x7f405c13f710 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.456+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f405c0abb90 con 0x7f405c13f710 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.456+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f40500019d0 con 0x7f405c13f710 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.456+0000 7f4043fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.456+0000 7f4043fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.456+0000 7f4043fff640 10 monclient(hunting): dump: 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:40.457 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:40.458 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:40.458 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:40.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.456+0000 7f4043fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:40.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1667482376 0 0) 0x7f405c0abb90 con 0x7f405c13f710 2024-03-25T18:28:40.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 >> v1:172.21.15.102:6790/0 conn(0x7f405c1430c0 legacy=0x7f405c1d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 >> v1:172.21.15.146:6789/0 conn(0x7f405c0a98f0 legacy=0x7f405c1383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:40.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:40.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 1 monclient: found mon.a 2024-03-25T18:28:40.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:40.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:40.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c138ad0 con 0x7f405c13f710 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4043fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 5 monclient: authenticate success, global_id 4356 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 10 monclient: _renew_subs 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c138d70 con 0x7f405c13f710 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 10 monclient: _renew_subs 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c1e32a0 con 0x7f405c13f710 2024-03-25T18:28:40.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.463+0000 7f4061b0e640 1 librados: init done 2024-03-25T18:28:40.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.464+0000 7f4061b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:40.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.464+0000 7f4061b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:40.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.464+0000 7f4061b0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:40.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.464+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c0627b0 con 0x7f405c13f710 2024-03-25T18:28:40.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.765+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4050001c90 con 0x7f405c13f710 2024-03-25T18:28:40.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.765+0000 7f4043fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:40.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.765+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4050003a60 con 0x7f405c13f710 2024-03-25T18:28:40.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.765+0000 7f4043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:40.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.765+0000 7f4043fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:40.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.765+0000 7f4043fff640 10 monclient: dump: 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:40.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.765+0000 7f4059ffb640 4 set_mon_vals no callback set 2024-03-25T18:28:40.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.766+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f4050003c80 con 0x7f405c13f710 2024-03-25T18:28:40.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.767+0000 7f405b7fe640 10 monclient: get_auth_request con 0x7f404806de90 auth_method 0 2024-03-25T18:28:40.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.767+0000 7f4043fff640 10 client.4356.objecter ms_handle_connect 0x7f404806de90 2024-03-25T18:28:40.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.775+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f4050054810 con 0x7f405c13f710 2024-03-25T18:28:40.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.775+0000 7f4043fff640 10 client.4356.objecter ms_dispatch 0x7f405c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:40.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.775+0000 7f4043fff640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:40.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.775+0000 7f4043fff640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:40.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.775+0000 7f4043fff640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-25T18:28:40.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.775+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4050050320 con 0x7f405c13f710 2024-03-25T18:28:40.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.775+0000 7f4043fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:40.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:40.775+0000 7f4043fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:41.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.036+0000 7f4061b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:41.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.036+0000 7f4061b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:41.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.036+0000 7f4061b0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:28:41.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.036+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f405c138f00 con 0x7f405c13f710 2024-03-25T18:28:41.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.067+0000 7f4043fff640 1 -- 172.21.15.102:0/4061937904 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f4050052250 con 0x7f405c13f710 2024-03-25T18:28:41.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.067+0000 7f4043fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:41.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.067+0000 7f4043fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:41.069 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:28:41.069 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 10 librados: watch_flush enter 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 10 librados: watch_flush exit 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 10 client.4356.objecter successfully canceled tick 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 >> v1:172.21.15.146:6800/3848252714 conn(0x7f404806de90 legacy=0x7f4048070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 10 monclient: shutdown 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 >> v1:172.21.15.102:6789/0 conn(0x7f405c13f710 legacy=0x7f405c1d7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 shutdown_connections 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.069+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 >> 172.21.15.102:0/4061937904 conn(0x7f405c0b7ea0 msgr2=0x7f405c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.070+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 shutdown_connections 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.070+0000 7f4061b0e640 1 -- 172.21.15.102:0/4061937904 wait complete. 2024-03-25T18:28:41.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:41.070+0000 7f4061b0e640 1 librados: shutdown 2024-03-25T18:28:41.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:47.889 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:48.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.005+0000 7fd426b85640 10 monclient: get_monmap_and_config 2024-03-25T18:28:48.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.005+0000 7fd426b85640 10 monclient: build_initial_monmap 2024-03-25T18:28:48.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.005+0000 7fd426b85640 10 monclient: monmap: 2024-03-25T18:28:48.006 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:28:48.006427+0000 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:28:48.006427+0000 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:28:48.007 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.006+0000 7fd426b85640 1 Processor -- start 2024-03-25T18:28:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.006+0000 7fd426b85640 1 -- start start 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.006+0000 7fd426b85640 10 monclient: init 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 10 monclient(hunting): picked mon.noname-c con 0x7fd4200a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 10 monclient(hunting): picked mon.noname-a con 0x7fd42013f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 10 monclient(hunting): picked mon.noname-b con 0x7fd4201430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd420142520 con 0x7fd42013f710 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4200b28a0 con 0x7fd4200a98f0 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4200b2a30 con 0x7fd4201430c0 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:48.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.007+0000 7fd426b85640 10 monclient(hunting): authenticate will time out at 7559.295588s 2024-03-25T18:28:48.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.018+0000 7fd425122640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd4201430c0 0x7fd42014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:50204/0 (socket says 172.21.15.102:50204) 2024-03-25T18:28:48.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.018+0000 7fd425122640 1 -- 172.21.15.102:0/859299344 learned_addr learned my addr 172.21.15.102:0/859299344 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:48.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.027+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528707210 0 0) 0x7fd4200b2a30 con 0x7fd4201430c0 2024-03-25T18:28:48.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.027+0000 7fd41f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:48.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.027+0000 7fd41f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:48.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.027+0000 7fd41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:48.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.027+0000 7fd41f7fe640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:28:48.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.027+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd410003520 con 0x7fd4201430c0 2024-03-25T18:28:48.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2557858588 0 0) 0x7fd410003520 con 0x7fd4201430c0 2024-03-25T18:28:48.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 >> v1:172.21.15.146:6789/0 conn(0x7fd4200a98f0 legacy=0x7fd4200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 >> v1:172.21.15.102:6789/0 conn(0x7fd42013f710 legacy=0x7fd4201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4200b44c0 con 0x7fd4201430c0 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd418001a10 con 0x7fd4201430c0 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd426b85640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd426b85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd41f7fe640 10 monclient: dump: 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:48.036 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:48.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.035+0000 7fd426b85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:48.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.041+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd418001b70 con 0x7fd4201430c0 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.041+0000 7fd41f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.041+0000 7fd41f7fe640 1 -- 172.21.15.102:0/859299344 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd418001fc0 con 0x7fd4201430c0 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.041+0000 7fd426b85640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.041+0000 7fd426b85640 4 set_mon_vals no callback set 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.041+0000 7fd41f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.041+0000 7fd41f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.041+0000 7fd41f7fe640 10 monclient: dump: 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:48.042 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:48.043 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:48.043 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:48.043 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:48.043 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:48.044 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.042+0000 7fd426b85640 10 monclient: shutdown 2024-03-25T18:28:48.044 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.042+0000 7fd426b85640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:48.044 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.042+0000 7fd426b85640 1 -- 172.21.15.102:0/859299344 >> v1:172.21.15.102:6790/0 conn(0x7fd4201430c0 legacy=0x7fd42014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.044 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.042+0000 7fd426b85640 1 -- 172.21.15.102:0/859299344 shutdown_connections 2024-03-25T18:28:48.044 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.042+0000 7fd426b85640 1 -- 172.21.15.102:0/859299344 >> 172.21.15.102:0/859299344 conn(0x7fd4200b7ea0 msgr2=0x7fd4200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:48.044 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.042+0000 7fd426b85640 1 -- 172.21.15.102:0/859299344 shutdown_connections 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.042+0000 7fd426b85640 1 -- 172.21.15.102:0/859299344 wait complete. 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 10 monclient: build_initial_monmap 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 10 monclient: monmap: 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:48.045 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 1 librados: starting msgr at 2024-03-25T18:28:48.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 1 librados: starting objecter 2024-03-25T18:28:48.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 1 Processor -- start 2024-03-25T18:28:48.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 1 -- start start 2024-03-25T18:28:48.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 1 librados: setting wanted keys 2024-03-25T18:28:48.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 1 librados: calling monclient init 2024-03-25T18:28:48.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.043+0000 7fd426b85640 10 monclient: init 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 10 monclient(hunting): picked mon.noname-b con 0x7fd4200a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 10 monclient(hunting): picked mon.noname-a con 0x7fd42013f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 10 monclient(hunting): picked mon.noname-c con 0x7fd4201430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd420138ed0 con 0x7fd42013f710 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd420139190 con 0x7fd4200a98f0 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4200abb00 con 0x7fd4201430c0 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.044+0000 7fd426b85640 10 monclient(hunting): authenticate will time out at 7559.332582s 2024-03-25T18:28:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.062+0000 7fd424921640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd4200a98f0 0x7fd4201364f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:33302/0 (socket says 172.21.15.102:33302) 2024-03-25T18:28:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.062+0000 7fd424921640 1 -- 172.21.15.102:0/2376461635 learned_addr learned my addr 172.21.15.102:0/2376461635 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.064+0000 7fd41d7fa640 10 client.?.objecter ms_handle_connect 0x7fd4200a98f0 2024-03-25T18:28:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.064+0000 7fd41d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:48.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.068+0000 7fd41d7fa640 10 client.?.objecter ms_handle_connect 0x7fd42013f710 2024-03-25T18:28:48.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.068+0000 7fd41d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:48.073 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.073+0000 7fd41d7fa640 10 client.?.objecter ms_handle_connect 0x7fd4201430c0 2024-03-25T18:28:48.073 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.073+0000 7fd41d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:48.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.075+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326866108 0 0) 0x7fd420139190 con 0x7fd4200a98f0 2024-03-25T18:28:48.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.075+0000 7fd41d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:48.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.075+0000 7fd41d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:48.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.075+0000 7fd41d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:48.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.075+0000 7fd41d7fa640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:28:48.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.075+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd404004eb0 con 0x7fd4200a98f0 2024-03-25T18:28:48.086 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.085+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946988233 0 0) 0x7fd420138ed0 con 0x7fd42013f710 2024-03-25T18:28:48.086 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.085+0000 7fd41d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:48.086 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.085+0000 7fd41d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:48.086 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.085+0000 7fd41d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:48.086 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.085+0000 7fd41d7fa640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:28:48.086 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.085+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd420139190 con 0x7fd42013f710 2024-03-25T18:28:48.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.096+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1856824534 0 0) 0x7fd404004eb0 con 0x7fd4200a98f0 2024-03-25T18:28:48.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.096+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd420138ed0 con 0x7fd4200a98f0 2024-03-25T18:28:48.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.096+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd4140019d0 con 0x7fd4200a98f0 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.096+0000 7fd41d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.096+0000 7fd41d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.096+0000 7fd41d7fa640 10 monclient(hunting): dump: 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:48.098 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:48.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.096+0000 7fd41d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2044325340 0 0) 0x7fd420138ed0 con 0x7fd4200a98f0 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 >> v1:172.21.15.102:6790/0 conn(0x7fd4201430c0 legacy=0x7fd4201387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 >> v1:172.21.15.102:6789/0 conn(0x7fd42013f710 legacy=0x7fd4200ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 1 monclient: found mon.b 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4200abd70 con 0x7fd4200a98f0 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd41d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:48.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:28:48.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 10 monclient: _renew_subs 2024-03-25T18:28:48.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:48.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4200ac050 con 0x7fd4200a98f0 2024-03-25T18:28:48.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:48.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 10 monclient: _renew_subs 2024-03-25T18:28:48.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:48.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4201e32a0 con 0x7fd4200a98f0 2024-03-25T18:28:48.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.106+0000 7fd426b85640 1 librados: init done 2024-03-25T18:28:48.109 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.107+0000 7fd426b85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:48.109 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.107+0000 7fd426b85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:48.109 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.107+0000 7fd426b85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:48.109 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.107+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4200627b0 con 0x7fd4200a98f0 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.376+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4140035f0 con 0x7fd4200a98f0 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.376+0000 7fd41d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.376+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd414003980 con 0x7fd4200a98f0 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.376+0000 7fd41d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.376+0000 7fd41d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.376+0000 7fd41d7fa640 10 monclient: dump: 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:48.377 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:48.378 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:48.378 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:48.378 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:48.378 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:48.378 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:48.378 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:48.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.376+0000 7fd41effd640 4 set_mon_vals no callback set 2024-03-25T18:28:48.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.377+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fd414003b60 con 0x7fd4200a98f0 2024-03-25T18:28:48.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.377+0000 7fd41ffff640 10 monclient: get_auth_request con 0x7fd40406dcc0 auth_method 0 2024-03-25T18:28:48.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.377+0000 7fd41d7fa640 10 client.4360.objecter ms_handle_connect 0x7fd40406dcc0 2024-03-25T18:28:48.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.382+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fd414054790 con 0x7fd4200a98f0 2024-03-25T18:28:48.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.382+0000 7fd41d7fa640 10 client.4360.objecter ms_dispatch 0x7fd420000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:48.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.382+0000 7fd41d7fa640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:48.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.382+0000 7fd41d7fa640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:48.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.382+0000 7fd41d7fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T18:28:48.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.383+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4140502a0 con 0x7fd4200a98f0 2024-03-25T18:28:48.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.383+0000 7fd41d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:48.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.383+0000 7fd41d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:48.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.588+0000 7fd426b85640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:48.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.588+0000 7fd426b85640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:48.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.588+0000 7fd426b85640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:28:48.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.588+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd42013df70 con 0x7fd4200a98f0 2024-03-25T18:28:48.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.637+0000 7fd41d7fa640 1 -- 172.21.15.102:0/2376461635 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fd4140521d0 con 0x7fd4200a98f0 2024-03-25T18:28:48.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.637+0000 7fd41d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:48.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.637+0000 7fd41d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:48.638 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:28:48.639 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 10 librados: watch_flush enter 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 10 librados: watch_flush exit 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 10 client.4360.objecter successfully canceled tick 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 >> v1:172.21.15.146:6800/3848252714 conn(0x7fd40406dcc0 legacy=0x7fd404070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 10 monclient: shutdown 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 >> v1:172.21.15.146:6789/0 conn(0x7fd4200a98f0 legacy=0x7fd4201364f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:48.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 shutdown_connections 2024-03-25T18:28:48.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 >> 172.21.15.102:0/2376461635 conn(0x7fd4200b7ea0 msgr2=0x7fd42013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:48.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 shutdown_connections 2024-03-25T18:28:48.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 1 -- 172.21.15.102:0/2376461635 wait complete. 2024-03-25T18:28:48.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:48.640+0000 7fd426b85640 1 librados: shutdown 2024-03-25T18:28:48.653 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:55.460 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:55.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.575+0000 7fb81a946640 10 monclient: get_monmap_and_config 2024-03-25T18:28:55.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.575+0000 7fb81a946640 10 monclient: build_initial_monmap 2024-03-25T18:28:55.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.575+0000 7fb81a946640 10 monclient: monmap: 2024-03-25T18:28:55.576 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:55.576 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:55.576 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:28:55.576265+0000 2024-03-25T18:28:55.576 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:28:55.576265+0000 2024-03-25T18:28:55.576 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:55.577 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:55.577 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:55.577 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:28:55.577 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:28:55.577 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.576+0000 7fb81a946640 1 Processor -- start 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.576+0000 7fb81a946640 1 -- start start 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.576+0000 7fb81a946640 10 monclient: init 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 10 monclient(hunting): picked mon.noname-c con 0x7fb8140a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 10 monclient(hunting): picked mon.noname-a con 0x7fb81413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 10 monclient(hunting): picked mon.noname-b con 0x7fb8141430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:55.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb814142520 con 0x7fb81413f710 2024-03-25T18:28:55.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8140b28a0 con 0x7fb8140a98f0 2024-03-25T18:28:55.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8140b2a30 con 0x7fb8141430c0 2024-03-25T18:28:55.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:55.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.577+0000 7fb81a946640 10 monclient(hunting): authenticate will time out at 7566.865090s 2024-03-25T18:28:55.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.587+0000 7fb813fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb8140a98f0 0x7fb8140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:42506/0 (socket says 172.21.15.102:42506) 2024-03-25T18:28:55.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.587+0000 7fb813fff640 1 -- 172.21.15.102:0/370168428 learned_addr learned my addr 172.21.15.102:0/370168428 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:55.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.600+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931539670 0 0) 0x7fb8140b28a0 con 0x7fb8140a98f0 2024-03-25T18:28:55.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.600+0000 7fb812ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:55.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.600+0000 7fb812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:55.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.600+0000 7fb812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:55.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.600+0000 7fb812ffd640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:28:55.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.600+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc003520 con 0x7fb8140a98f0 2024-03-25T18:28:55.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.610+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568911095 0 0) 0x7fb814142520 con 0x7fb81413f710 2024-03-25T18:28:55.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.610+0000 7fb812ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:55.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.610+0000 7fb812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:55.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.610+0000 7fb812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:55.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.610+0000 7fb812ffd640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:28:55.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.610+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8140b28a0 con 0x7fb81413f710 2024-03-25T18:28:55.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2617846932 0 0) 0x7fb7fc003520 con 0x7fb8140a98f0 2024-03-25T18:28:55.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 >> v1:172.21.15.102:6790/0 conn(0x7fb8141430c0 legacy=0x7fb81414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 >> v1:172.21.15.102:6789/0 conn(0x7fb81413f710 legacy=0x7fb8141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:55.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 1 monclient: found mon.noname-c 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8140b44c0 con 0x7fb8140a98f0 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb804001690 con 0x7fb8140a98f0 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb81a946640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb81a946640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb812ffd640 10 monclient: dump: 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.619 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:55.620 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:55.620 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:55.620 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:55.620 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:55.620 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:55.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.617+0000 7fb81a946640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.632+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8040017f0 con 0x7fb8140a98f0 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.632+0000 7fb812ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.632+0000 7fb812ffd640 1 -- 172.21.15.102:0/370168428 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb804001c00 con 0x7fb8140a98f0 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.632+0000 7fb81a946640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.632+0000 7fb81a946640 4 set_mon_vals no callback set 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.632+0000 7fb812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.632+0000 7fb812ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.632+0000 7fb812ffd640 10 monclient: dump: 2024-03-25T18:28:55.633 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:55.634 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:55.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.633+0000 7fb81a946640 10 monclient: shutdown 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.633+0000 7fb81a946640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.633+0000 7fb81a946640 1 -- 172.21.15.102:0/370168428 >> v1:172.21.15.146:6789/0 conn(0x7fb8140a98f0 legacy=0x7fb8140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.633+0000 7fb81a946640 1 -- 172.21.15.102:0/370168428 shutdown_connections 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.633+0000 7fb81a946640 1 -- 172.21.15.102:0/370168428 >> 172.21.15.102:0/370168428 conn(0x7fb8140b7ea0 msgr2=0x7fb8140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.633+0000 7fb81a946640 1 -- 172.21.15.102:0/370168428 shutdown_connections 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.633+0000 7fb81a946640 1 -- 172.21.15.102:0/370168428 wait complete. 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.634+0000 7fb81a946640 10 monclient: build_initial_monmap 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.634+0000 7fb81a946640 10 monclient: monmap: 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:55.636 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.634+0000 7fb81a946640 1 librados: starting msgr at 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.634+0000 7fb81a946640 1 librados: starting objecter 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 1 Processor -- start 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 1 -- start start 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 1 librados: setting wanted keys 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 1 librados: calling monclient init 2024-03-25T18:28:55.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 10 monclient: init 2024-03-25T18:28:55.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:55.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:55.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 10 monclient(hunting): picked mon.noname-c con 0x7fb8140a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:28:55.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.635+0000 7fb81a946640 10 monclient(hunting): picked mon.noname-b con 0x7fb81413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:28:55.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.636+0000 7fb81a946640 10 monclient(hunting): picked mon.noname-a con 0x7fb8141430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:28:55.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.636+0000 7fb81a946640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8141d7380 con 0x7fb8141430c0 2024-03-25T18:28:55.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.636+0000 7fb81a946640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb814139840 con 0x7fb81413f710 2024-03-25T18:28:55.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.636+0000 7fb81a946640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8141d8200 con 0x7fb8140a98f0 2024-03-25T18:28:55.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.636+0000 7fb81a946640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:55.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.636+0000 7fb81a946640 10 monclient(hunting): authenticate will time out at 7566.924040s 2024-03-25T18:28:55.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.646+0000 7fb8137fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb81413f710 0x7fb8141da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:42550/0 (socket says 172.21.15.102:42550) 2024-03-25T18:28:55.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.646+0000 7fb8137fe640 1 -- 172.21.15.102:0/82791922 learned_addr learned my addr 172.21.15.102:0/82791922 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:28:55.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.647+0000 7fb810ff9640 10 client.?.objecter ms_handle_connect 0x7fb81413f710 2024-03-25T18:28:55.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.647+0000 7fb810ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:55.651 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.651+0000 7fb810ff9640 10 client.?.objecter ms_handle_connect 0x7fb8141430c0 2024-03-25T18:28:55.651 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.651+0000 7fb810ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:55.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.655+0000 7fb810ff9640 10 client.?.objecter ms_handle_connect 0x7fb8140a98f0 2024-03-25T18:28:55.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.655+0000 7fb810ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:55.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.656+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704798967 0 0) 0x7fb814139840 con 0x7fb81413f710 2024-03-25T18:28:55.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.656+0000 7fb810ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:55.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.656+0000 7fb810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:55.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.656+0000 7fb810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:55.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.656+0000 7fb810ff9640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:28:55.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.656+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb800004eb0 con 0x7fb81413f710 2024-03-25T18:28:55.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.662+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278242887 0 0) 0x7fb8141d8200 con 0x7fb8140a98f0 2024-03-25T18:28:55.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.662+0000 7fb810ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:55.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.662+0000 7fb810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.662+0000 7fb810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.662+0000 7fb810ff9640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:28:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.662+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb814139840 con 0x7fb8140a98f0 2024-03-25T18:28:55.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.665+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024743304 0 0) 0x7fb8141d7380 con 0x7fb8141430c0 2024-03-25T18:28:55.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.665+0000 7fb810ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:55.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.665+0000 7fb810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:55.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.665+0000 7fb810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:55.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.665+0000 7fb810ff9640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:28:55.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.665+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8141d8200 con 0x7fb8141430c0 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.672+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1743252611 0 0) 0x7fb814139840 con 0x7fb8140a98f0 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.672+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8141d7380 con 0x7fb8140a98f0 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.672+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb804001b40 con 0x7fb8140a98f0 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.672+0000 7fb810ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.672+0000 7fb810ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.672+0000 7fb810ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.673 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:55.674 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:55.674 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:55.674 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:55.674 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:55.674 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:55.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.672+0000 7fb810ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:55.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.677+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3937196965 0 0) 0x7fb8141d8200 con 0x7fb8141430c0 2024-03-25T18:28:55.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.677+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb814139840 con 0x7fb8141430c0 2024-03-25T18:28:55.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.677+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb80c001a60 con 0x7fb8141430c0 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.677+0000 7fb810ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.677+0000 7fb810ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.677+0000 7fb810ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:55.678 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:55.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3463606197 0 0) 0x7fb8141d7380 con 0x7fb8140a98f0 2024-03-25T18:28:55.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 >> v1:172.21.15.146:6789/0 conn(0x7fb81413f710 legacy=0x7fb8141da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 >> v1:172.21.15.102:6789/0 conn(0x7fb8141430c0 legacy=0x7fb8141daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 1 monclient: found mon.c 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8141d75d0 con 0x7fb8140a98f0 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb810ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 5 monclient: authenticate success, global_id 4349 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 10 monclient: _renew_subs 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8141db6a0 con 0x7fb8140a98f0 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 10 monclient: _renew_subs 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb814137910 con 0x7fb8140a98f0 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.687+0000 7fb81a946640 1 librados: init done 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.688+0000 7fb81a946640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.688+0000 7fb81a946640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.688+0000 7fb81a946640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:55.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.688+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8140627b0 con 0x7fb8140a98f0 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.992+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb804004a90 con 0x7fb8140a98f0 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.992+0000 7fb810ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.992+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb804004da0 con 0x7fb8140a98f0 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.992+0000 7fb810ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.992+0000 7fb810ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.992+0000 7fb810ff9640 10 monclient: dump: 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:28:55.993 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.994 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:28:55.994 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:55.994 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:28:55.994 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:28:55.994 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:28:55.994 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:28:55.994 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:28:55.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.992+0000 7fb8127fc640 4 set_mon_vals no callback set 2024-03-25T18:28:55.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.993+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fb80401ad70 con 0x7fb8140a98f0 2024-03-25T18:28:55.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.994+0000 7fb8137fe640 10 monclient: get_auth_request con 0x7fb80006df80 auth_method 0 2024-03-25T18:28:55.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:55.994+0000 7fb810ff9640 10 client.4349.objecter ms_handle_connect 0x7fb80006df80 2024-03-25T18:28:56.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.001+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fb8040548a0 con 0x7fb8140a98f0 2024-03-25T18:28:56.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.001+0000 7fb810ff9640 10 client.4349.objecter ms_dispatch 0x7fb814000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:56.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.001+0000 7fb810ff9640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:56.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.001+0000 7fb810ff9640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:56.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.001+0000 7fb810ff9640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-25T18:28:56.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.001+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8040503b0 con 0x7fb8140a98f0 2024-03-25T18:28:56.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.001+0000 7fb810ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:56.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.001+0000 7fb810ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:56.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.253+0000 7fb81a946640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:56.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.253+0000 7fb81a946640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:56.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.253+0000 7fb81a946640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:28:56.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.253+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8141d7810 con 0x7fb8140a98f0 2024-03-25T18:28:56.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.275+0000 7fb810ff9640 1 -- 172.21.15.102:0/82791922 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fb8040522e0 con 0x7fb8140a98f0 2024-03-25T18:28:56.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.275+0000 7fb810ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:56.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.275+0000 7fb810ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:56.277 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:28:56.277 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:56.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 10 librados: watch_flush enter 2024-03-25T18:28:56.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 10 librados: watch_flush exit 2024-03-25T18:28:56.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 10 client.4349.objecter successfully canceled tick 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 >> v1:172.21.15.146:6800/3848252714 conn(0x7fb80006df80 legacy=0x7fb800070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 10 monclient: shutdown 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 >> v1:172.21.15.102:6790/0 conn(0x7fb8140a98f0 legacy=0x7fb8141d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 shutdown_connections 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 >> 172.21.15.102:0/82791922 conn(0x7fb8140b7ea0 msgr2=0x7fb814074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 shutdown_connections 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.278+0000 7fb81a946640 1 -- 172.21.15.102:0/82791922 wait complete. 2024-03-25T18:28:56.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:28:56.279+0000 7fb81a946640 1 librados: shutdown 2024-03-25T18:28:56.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:03.097 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:03.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.200+0000 7fc47971c640 10 monclient: get_monmap_and_config 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.200+0000 7fc47971c640 10 monclient: build_initial_monmap 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.200+0000 7fc47971c640 10 monclient: monmap: 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:29:03.201878+0000 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:29:03.201878+0000 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:29:03.202 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:03.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.200+0000 7fc47971c640 1 Processor -- start 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.200+0000 7fc47971c640 1 -- start start 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 10 monclient: init 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 10 monclient(hunting): picked mon.noname-a con 0x7fc47406ab80 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 10 monclient(hunting): picked mon.noname-b con 0x7fc47406af60 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 10 monclient(hunting): picked mon.noname-c con 0x7fc47413f990 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740afa60 con 0x7fc47406ab80 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740afce0 con 0x7fc47413f990 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740b1150 con 0x7fc47406af60 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:03.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.201+0000 7fc47971c640 10 monclient(hunting): authenticate will time out at 7574.490628s 2024-03-25T18:29:03.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.209+0000 7fc4737fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc47413f990 0x7fc47414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:59458/0 (socket says 172.21.15.102:59458) 2024-03-25T18:29:03.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.209+0000 7fc4737fe640 1 -- 172.21.15.102:0/1376933934 learned_addr learned my addr 172.21.15.102:0/1376933934 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:03.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.221+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252612941 0 0) 0x7fc4740afce0 con 0x7fc47413f990 2024-03-25T18:29:03.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.221+0000 7fc471ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:03.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.221+0000 7fc471ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:03.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.221+0000 7fc471ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:03.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.221+0000 7fc471ffb640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:29:03.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.221+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c003520 con 0x7fc47413f990 2024-03-25T18:29:03.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2249602905 0 0) 0x7fc45c003520 con 0x7fc47413f990 2024-03-25T18:29:03.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 >> v1:172.21.15.102:6790/0 conn(0x7fc47406af60 legacy=0x7fc4741458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:03.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 >> v1:172.21.15.102:6789/0 conn(0x7fc47406ab80 legacy=0x7fc4740724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 1 monclient: found mon.noname-c 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740aff20 con 0x7fc47413f990 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc46c001a10 con 0x7fc47413f990 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc471ffb640 10 monclient: dump: 2024-03-25T18:29:03.229 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc47971c640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:29:03.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.226+0000 7fc47971c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.250+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc46c001b70 con 0x7fc47413f990 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.250+0000 7fc471ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.250+0000 7fc471ffb640 1 -- 172.21.15.102:0/1376933934 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc46c001f80 con 0x7fc47413f990 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.250+0000 7fc471ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.250+0000 7fc471ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.250+0000 7fc471ffb640 10 monclient: dump: 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.252 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:03.253 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:03.253 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:03.253 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:03.253 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:03.253 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:03.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.250+0000 7fc47971c640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:03.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.250+0000 7fc47971c640 4 set_mon_vals no callback set 2024-03-25T18:29:03.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 10 monclient: shutdown 2024-03-25T18:29:03.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:03.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 1 -- 172.21.15.102:0/1376933934 >> v1:172.21.15.146:6789/0 conn(0x7fc47413f990 legacy=0x7fc47414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:03.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 1 -- 172.21.15.102:0/1376933934 shutdown_connections 2024-03-25T18:29:03.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 1 -- 172.21.15.102:0/1376933934 >> 172.21.15.102:0/1376933934 conn(0x7fc47406c830 msgr2=0x7fc4740b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:03.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 1 -- 172.21.15.102:0/1376933934 shutdown_connections 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 1 -- 172.21.15.102:0/1376933934 wait complete. 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 10 monclient: build_initial_monmap 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.251+0000 7fc47971c640 10 monclient: monmap: 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:03.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.252+0000 7fc47971c640 1 librados: starting msgr at 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.252+0000 7fc47971c640 1 librados: starting objecter 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.252+0000 7fc47971c640 1 Processor -- start 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.252+0000 7fc47971c640 1 -- start start 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.252+0000 7fc47971c640 1 librados: setting wanted keys 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.252+0000 7fc47971c640 1 librados: calling monclient init 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.252+0000 7fc47971c640 10 monclient: init 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 10 monclient(hunting): picked mon.noname-b con 0x7fc47406ab80 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 10 monclient(hunting): picked mon.noname-a con 0x7fc47413f990 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 10 monclient(hunting): picked mon.noname-c con 0x7fc47414e7f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4741db350 con 0x7fc47413f990 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4741db4e0 con 0x7fc47406ab80 2024-03-25T18:29:03.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4741db670 con 0x7fc47414e7f0 2024-03-25T18:29:03.258 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:03.258 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.253+0000 7fc47971c640 10 monclient(hunting): authenticate will time out at 7574.542266s 2024-03-25T18:29:03.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.264+0000 7fc472ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc47406ab80 0x7fc4741dac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:59514/0 (socket says 172.21.15.102:59514) 2024-03-25T18:29:03.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.264+0000 7fc472ffd640 1 -- 172.21.15.102:0/2978301012 learned_addr learned my addr 172.21.15.102:0/2978301012 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:03.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.268+0000 7fc45b7fe640 10 client.?.objecter ms_handle_connect 0x7fc47406ab80 2024-03-25T18:29:03.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.268+0000 7fc45b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:03.272 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.271+0000 7fc45b7fe640 10 client.?.objecter ms_handle_connect 0x7fc47413f990 2024-03-25T18:29:03.272 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.271+0000 7fc45b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:03.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.277+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688353121 0 0) 0x7fc4741db4e0 con 0x7fc47406ab80 2024-03-25T18:29:03.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.277+0000 7fc45b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:03.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.277+0000 7fc45b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:03.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.277+0000 7fc45b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:03.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.277+0000 7fc45b7fe640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:29:03.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.277+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc460004eb0 con 0x7fc47406ab80 2024-03-25T18:29:03.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.280+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995435676 0 0) 0x7fc4741db350 con 0x7fc47413f990 2024-03-25T18:29:03.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.280+0000 7fc45b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:03.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.280+0000 7fc45b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:03.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.280+0000 7fc45b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:03.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.280+0000 7fc45b7fe640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:29:03.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.280+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4741db4e0 con 0x7fc47413f990 2024-03-25T18:29:03.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.286+0000 7fc45b7fe640 10 client.?.objecter ms_handle_connect 0x7fc47414e7f0 2024-03-25T18:29:03.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.286+0000 7fc45b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:03.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.299+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770748681 0 0) 0x7fc4741db670 con 0x7fc47414e7f0 2024-03-25T18:29:03.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.299+0000 7fc45b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:03.302 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.299+0000 7fc45b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:03.302 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.299+0000 7fc45b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:03.302 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.299+0000 7fc45b7fe640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:29:03.302 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.299+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4741db350 con 0x7fc47414e7f0 2024-03-25T18:29:03.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.301+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1810566923 0 0) 0x7fc4741db4e0 con 0x7fc47413f990 2024-03-25T18:29:03.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.301+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4741db670 con 0x7fc47413f990 2024-03-25T18:29:03.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.301+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc468001930 con 0x7fc47413f990 2024-03-25T18:29:03.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.301+0000 7fc45b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:03.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.301+0000 7fc45b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:03.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.301+0000 7fc45b7fe640 10 monclient(hunting): dump: 2024-03-25T18:29:03.305 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:03.305 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:03.305 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.305 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.306 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:03.306 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:03.306 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:03.306 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:03.306 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:03.306 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:03.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.301+0000 7fc45b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.313+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2690728000 0 0) 0x7fc4741db350 con 0x7fc47414e7f0 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.313+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4741db4e0 con 0x7fc47414e7f0 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.313+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc46c001e80 con 0x7fc47414e7f0 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.313+0000 7fc45b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.313+0000 7fc45b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.313+0000 7fc45b7fe640 10 monclient(hunting): dump: 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.315 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:03.316 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:03.316 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:03.316 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:03.316 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:03.316 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2934245204 0 0) 0x7fc4741db4e0 con 0x7fc47414e7f0 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 >> v1:172.21.15.146:6789/0 conn(0x7fc47406ab80 legacy=0x7fc4741dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 >> v1:172.21.15.102:6789/0 conn(0x7fc47413f990 legacy=0x7fc47413da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 1 monclient: found mon.c 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4741e5fc0 con 0x7fc47414e7f0 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc45b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 5 monclient: authenticate success, global_id 4355 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 10 monclient: _renew_subs 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4741e2f90 con 0x7fc47414e7f0 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 10 monclient: _renew_subs 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4741e3470 con 0x7fc47414e7f0 2024-03-25T18:29:03.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.321+0000 7fc47971c640 1 librados: init done 2024-03-25T18:29:03.325 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.322+0000 7fc47971c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:03.325 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.322+0000 7fc47971c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:03.325 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.322+0000 7fc47971c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:03.325 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.322+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc474074d90 con 0x7fc47414e7f0 2024-03-25T18:29:03.606 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:19.841 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.618+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc46c001fe0 con 0x7fc47414e7f0 2024-03-25T18:29:03.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.618+0000 7fc45b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:03.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.618+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc46c004fb0 con 0x7fc47414e7f0 2024-03-25T18:29:03.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.618+0000 7fc45b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:03.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.618+0000 7fc45b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.618+0000 7fc45b7fe640 10 monclient: dump: 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:03.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.618+0000 7fc4717fa640 4 set_mon_vals no callback set 2024-03-25T18:29:03.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.619+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fc46c01af80 con 0x7fc47414e7f0 2024-03-25T18:29:03.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.620+0000 7fc472ffd640 10 monclient: get_auth_request con 0x7fc46006df30 auth_method 0 2024-03-25T18:29:03.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.620+0000 7fc45b7fe640 10 client.4355.objecter ms_handle_connect 0x7fc46006df30 2024-03-25T18:29:03.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.624+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fc46c054ab0 con 0x7fc47414e7f0 2024-03-25T18:29:03.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.624+0000 7fc45b7fe640 10 client.4355.objecter ms_dispatch 0x7fc474000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:03.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.624+0000 7fc45b7fe640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:03.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.624+0000 7fc45b7fe640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:03.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.624+0000 7fc45b7fe640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-25T18:29:03.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.624+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc46c0505c0 con 0x7fc47414e7f0 2024-03-25T18:29:03.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.624+0000 7fc45b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:03.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.624+0000 7fc45b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:03.628 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:19.864 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.656 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:19.869 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.657 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:19.892 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.660 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:19.896 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.671 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:19.884 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.677 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:19.914 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.715 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:19.928 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.730 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:19.965 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:03.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.842+0000 7fc47971c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:03.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.842+0000 7fc47971c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:03.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.842+0000 7fc47971c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:03.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.842+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4740627b0 con 0x7fc47414e7f0 2024-03-25T18:29:03.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.873+0000 7fc45b7fe640 1 -- 172.21.15.102:0/2978301012 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fc46c0524f0 con 0x7fc47414e7f0 2024-03-25T18:29:03.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.873+0000 7fc45b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:03.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.873+0000 7fc45b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:03.875 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:29:03.875 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:03.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 10 librados: watch_flush enter 2024-03-25T18:29:03.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 10 librados: watch_flush exit 2024-03-25T18:29:03.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-25T18:29:03.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 10 client.4355.objecter successfully canceled tick 2024-03-25T18:29:03.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 >> v1:172.21.15.146:6800/3848252714 conn(0x7fc46006df30 legacy=0x7fc4600703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:03.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 10 monclient: shutdown 2024-03-25T18:29:03.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:03.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 >> v1:172.21.15.102:6790/0 conn(0x7fc47414e7f0 legacy=0x7fc47413e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:03.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 shutdown_connections 2024-03-25T18:29:03.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 >> 172.21.15.102:0/2978301012 conn(0x7fc47406c830 msgr2=0x7fc4740b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:03.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 shutdown_connections 2024-03-25T18:29:03.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 1 -- 172.21.15.102:0/2978301012 wait complete. 2024-03-25T18:29:03.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:03.875+0000 7fc47971c640 1 librados: shutdown 2024-03-25T18:29:03.888 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:10.695 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:10.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.810+0000 7f64184cf640 10 monclient: get_monmap_and_config 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.810+0000 7f64184cf640 10 monclient: build_initial_monmap 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.810+0000 7f64184cf640 10 monclient: monmap: 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:29:10.812012+0000 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:29:10.812012+0000 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:29:10.812 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:10.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.811+0000 7f64184cf640 1 Processor -- start 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.811+0000 7f64184cf640 1 -- start start 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.811+0000 7f64184cf640 10 monclient: init 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.811+0000 7f64184cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.811+0000 7f64184cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.811+0000 7f64184cf640 10 monclient(hunting): picked mon.noname-a con 0x7f641013fc60 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.811+0000 7f64184cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6410070570 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.811+0000 7f64184cf640 10 monclient(hunting): picked mon.noname-c con 0x7f64100727a0 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.812+0000 7f64184cf640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641013c9f0 con 0x7f641013fc60 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.812+0000 7f64184cf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641006fac0 con 0x7f64100727a0 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.812+0000 7f64184cf640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64100715d0 con 0x7f6410070570 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.812+0000 7f64184cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:10.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.812+0000 7f64184cf640 10 monclient(hunting): authenticate will time out at 7582.100959s 2024-03-25T18:29:10.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.819+0000 7f6416a6c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f64100727a0 0x7f641014a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:36194/0 (socket says 172.21.15.102:36194) 2024-03-25T18:29:10.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.819+0000 7f6416a6c640 1 -- 172.21.15.102:0/217829174 learned_addr learned my addr 172.21.15.102:0/217829174 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:10.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.825+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239953379 0 0) 0x7f64100715d0 con 0x7f6410070570 2024-03-25T18:29:10.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.825+0000 7f6415269640 10 monclient(hunting): _negotiate 2024-03-25T18:29:10.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.825+0000 7f6415269640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:10.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.825+0000 7f6415269640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:10.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.825+0000 7f6415269640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:29:10.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.825+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63fc003520 con 0x7f6410070570 2024-03-25T18:29:10.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.826+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404226891 0 0) 0x7f641006fac0 con 0x7f64100727a0 2024-03-25T18:29:10.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.826+0000 7f6415269640 10 monclient(hunting): _negotiate 2024-03-25T18:29:10.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.826+0000 7f6415269640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:10.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.826+0000 7f6415269640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:10.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.826+0000 7f6415269640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:29:10.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.826+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64100715d0 con 0x7f64100727a0 2024-03-25T18:29:10.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1504477874 0 0) 0x7f63fc003520 con 0x7f6410070570 2024-03-25T18:29:10.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 >> v1:172.21.15.146:6789/0 conn(0x7f64100727a0 legacy=0x7f641014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 >> v1:172.21.15.102:6789/0 conn(0x7f641013fc60 legacy=0x7f641006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 10 monclient: _finish_hunting 0 2024-03-25T18:29:10.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 1 monclient: found mon.noname-b 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64100718b0 con 0x7f6410070570 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 10 monclient: _finish_auth 0 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 10 monclient: _check_auth_tickets 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6400001690 con 0x7f6410070570 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f6415269640 10 monclient: dump: 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:10.834 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:10.835 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:10.835 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:10.835 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:10.835 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:10.835 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:10.835 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:10.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f64184cf640 5 monclient: authenticate success, global_id 4358 2024-03-25T18:29:10.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.831+0000 7f64184cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.838+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64000017f0 con 0x7f6410070570 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.838+0000 7f6415269640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.838+0000 7f6415269640 1 -- 172.21.15.102:0/217829174 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6400001c00 con 0x7f6410070570 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.838+0000 7f64184cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.838+0000 7f64184cf640 4 set_mon_vals no callback set 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.838+0000 7f6415269640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.838+0000 7f6415269640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.838+0000 7f6415269640 10 monclient: dump: 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:10.840 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:10.841 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:10.841 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:10.841 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:10.841 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:10.841 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:10.841 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:10.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.839+0000 7f64184cf640 10 monclient: shutdown 2024-03-25T18:29:10.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.839+0000 7f64184cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:10.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.839+0000 7f64184cf640 1 -- 172.21.15.102:0/217829174 >> v1:172.21.15.102:6790/0 conn(0x7f6410070570 legacy=0x7f6410072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.839+0000 7f64184cf640 1 -- 172.21.15.102:0/217829174 shutdown_connections 2024-03-25T18:29:10.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.839+0000 7f64184cf640 1 -- 172.21.15.102:0/217829174 >> 172.21.15.102:0/217829174 conn(0x7f64100b4a80 msgr2=0x7f64100b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:10.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.839+0000 7f64184cf640 1 -- 172.21.15.102:0/217829174 shutdown_connections 2024-03-25T18:29:10.846 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.839+0000 7f64184cf640 1 -- 172.21.15.102:0/217829174 wait complete. 2024-03-25T18:29:10.846 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 10 monclient: build_initial_monmap 2024-03-25T18:29:10.846 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 10 monclient: monmap: 2024-03-25T18:29:10.846 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:10.846 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:10.847 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:29:10.847 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:29:10.847 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:10.847 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:29:10.847 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:10.847 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:29:10.848 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:29:10.848 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:10.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 1 librados: starting msgr at 2024-03-25T18:29:10.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 1 librados: starting objecter 2024-03-25T18:29:10.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 1 Processor -- start 2024-03-25T18:29:10.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 1 -- start start 2024-03-25T18:29:10.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 1 librados: setting wanted keys 2024-03-25T18:29:10.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 1 librados: calling monclient init 2024-03-25T18:29:10.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.840+0000 7f64184cf640 10 monclient: init 2024-03-25T18:29:10.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:10.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:10.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6410070570 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:10.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 10 monclient(hunting): picked mon.noname-c con 0x7f64100727a0 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:10.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 10 monclient(hunting): picked mon.noname-b con 0x7f641013fc60 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:10.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6410139f90 con 0x7f6410070570 2024-03-25T18:29:10.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6410140bf0 con 0x7f641013fc60 2024-03-25T18:29:10.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6410139bc0 con 0x7f64100727a0 2024-03-25T18:29:10.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:10.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.841+0000 7f64184cf640 10 monclient(hunting): authenticate will time out at 7582.130359s 2024-03-25T18:29:10.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.844+0000 7f6416a6c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f641013fc60 0x7f64101e24b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:36210/0 (socket says 172.21.15.102:36210) 2024-03-25T18:29:10.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.844+0000 7f6416a6c640 1 -- 172.21.15.102:0/1342764509 learned_addr learned my addr 172.21.15.102:0/1342764509 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:10.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.848+0000 7f6406ffd640 10 client.?.objecter ms_handle_connect 0x7f641013fc60 2024-03-25T18:29:10.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.848+0000 7f6406ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:10.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.859+0000 7f6406ffd640 10 client.?.objecter ms_handle_connect 0x7f6410070570 2024-03-25T18:29:10.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.859+0000 7f6406ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:10.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.862+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445629589 0 0) 0x7f6410140bf0 con 0x7f641013fc60 2024-03-25T18:29:10.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.862+0000 7f6406ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:10.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.862+0000 7f6406ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:10.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.862+0000 7f6406ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:10.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.862+0000 7f6406ffd640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:29:10.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.862+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f4004eb0 con 0x7f641013fc60 2024-03-25T18:29:10.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.872+0000 7f6406ffd640 10 client.?.objecter ms_handle_connect 0x7f64100727a0 2024-03-25T18:29:10.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.872+0000 7f6406ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:10.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.883+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639631351 0 0) 0x7f6410139f90 con 0x7f6410070570 2024-03-25T18:29:10.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.883+0000 7f6406ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:10.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.883+0000 7f6406ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:10.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.883+0000 7f6406ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:10.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.883+0000 7f6406ffd640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:29:10.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.884+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6410140bf0 con 0x7f6410070570 2024-03-25T18:29:10.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.895+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661816445 0 0) 0x7f6410139bc0 con 0x7f64100727a0 2024-03-25T18:29:10.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.895+0000 7f6406ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:10.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.895+0000 7f6406ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:10.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.895+0000 7f6406ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:10.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.895+0000 7f6406ffd640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:29:10.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.895+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6410139f90 con 0x7f64100727a0 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.899+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3323362407 0 0) 0x7f63f4004eb0 con 0x7f641013fc60 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.899+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6410139bc0 con 0x7f641013fc60 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.899+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6408001cd0 con 0x7f641013fc60 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.899+0000 7f6406ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.899+0000 7f6406ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.899+0000 7f6406ffd640 10 monclient(hunting): dump: 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:10.901 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:10.902 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:10.902 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:10.902 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:10.902 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.899+0000 7f6406ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:10.906 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655189589 0 0) 0x7f6410139bc0 con 0x7f641013fc60 2024-03-25T18:29:10.906 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 >> v1:172.21.15.102:6790/0 conn(0x7f64100727a0 legacy=0x7f64101daea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.906 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 >> v1:172.21.15.102:6789/0 conn(0x7f6410070570 legacy=0x7f64101394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:10.906 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:10.906 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 1 monclient: found mon.b 2024-03-25T18:29:10.906 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:10.906 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64100adaf0 con 0x7f641013fc60 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 10 monclient: _finish_auth 0 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f6406ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 10 monclient: _renew_subs 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64100addc0 con 0x7f641013fc60 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 10 monclient: _renew_subs 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64101db790 con 0x7f641013fc60 2024-03-25T18:29:10.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.904+0000 7f64184cf640 1 librados: init done 2024-03-25T18:29:10.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.905+0000 7f64184cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:10.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.905+0000 7f64184cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:10.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.905+0000 7f64184cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:29:10.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:10.905+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6410139da0 con 0x7f641013fc60 2024-03-25T18:29:11.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.186+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6408004b60 con 0x7f641013fc60 2024-03-25T18:29:11.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.186+0000 7f6406ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:11.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.186+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6408004e70 con 0x7f641013fc60 2024-03-25T18:29:11.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.186+0000 7f6406ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:11.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.186+0000 7f6406ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:11.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.186+0000 7f6406ffd640 10 monclient: dump: 2024-03-25T18:29:11.189 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:11.189 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:11.190 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:11.190 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:11.190 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:11.190 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:11.190 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:11.190 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:11.191 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:11.191 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:11.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.186+0000 7f6414a68640 4 set_mon_vals no callback set 2024-03-25T18:29:11.192 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.187+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f640801ae40 con 0x7f641013fc60 2024-03-25T18:29:11.192 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.188+0000 7f641626b640 10 monclient: get_auth_request con 0x7f63f406dda0 auth_method 0 2024-03-25T18:29:11.192 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.188+0000 7f6406ffd640 10 client.4378.objecter ms_handle_connect 0x7f63f406dda0 2024-03-25T18:29:11.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.200+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f6408054b60 con 0x7f641013fc60 2024-03-25T18:29:11.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.200+0000 7f6406ffd640 10 client.4378.objecter ms_dispatch 0x7f6410000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:11.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.200+0000 7f6406ffd640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:11.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.200+0000 7f6406ffd640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:11.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.200+0000 7f6406ffd640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T18:29:11.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.202+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64080504a0 con 0x7f641013fc60 2024-03-25T18:29:11.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.202+0000 7f6406ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:11.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.202+0000 7f6406ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:11.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.417+0000 7f64184cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:11.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.417+0000 7f64184cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:11.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.417+0000 7f64184cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:29:11.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.417+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64100627b0 con 0x7f641013fc60 2024-03-25T18:29:11.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.435+0000 7f6406ffd640 1 -- 172.21.15.102:0/1342764509 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f64080525a0 con 0x7f641013fc60 2024-03-25T18:29:11.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.435+0000 7f6406ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:11.437 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.435+0000 7f6406ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:11.437 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:29:11.437 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:11.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.437+0000 7f64184cf640 10 librados: watch_flush enter 2024-03-25T18:29:11.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.437+0000 7f64184cf640 10 librados: watch_flush exit 2024-03-25T18:29:11.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.437+0000 7f64184cf640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T18:29:11.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.437+0000 7f64184cf640 10 client.4378.objecter successfully canceled tick 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 >> v1:172.21.15.146:6800/3848252714 conn(0x7f63f406dda0 legacy=0x7f63f4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 10 monclient: shutdown 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 >> v1:172.21.15.146:6789/0 conn(0x7f641013fc60 legacy=0x7f64101e24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 shutdown_connections 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 >> 172.21.15.102:0/1342764509 conn(0x7f64100b4a80 msgr2=0x7f6410074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 shutdown_connections 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 1 -- 172.21.15.102:0/1342764509 wait complete. 2024-03-25T18:29:11.441 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:11.438+0000 7f64184cf640 1 librados: shutdown 2024-03-25T18:29:11.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:18.259 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:18.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.368+0000 7f6d51a93640 10 monclient: get_monmap_and_config 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.368+0000 7f6d51a93640 10 monclient: build_initial_monmap 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.368+0000 7f6d51a93640 10 monclient: monmap: 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:29:18.369783+0000 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:29:18.369783+0000 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:29:18.370 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.369+0000 7f6d51a93640 1 Processor -- start 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.369+0000 7f6d51a93640 1 -- start start 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.369+0000 7f6d51a93640 10 monclient: init 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.369+0000 7f6d51a93640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.369+0000 7f6d51a93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.369+0000 7f6d51a93640 10 monclient(hunting): picked mon.noname-c con 0x7f6d4c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.370+0000 7f6d51a93640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.370+0000 7f6d51a93640 10 monclient(hunting): picked mon.noname-b con 0x7f6d4c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.370+0000 7f6d51a93640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c142520 con 0x7f6d4c13f710 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.370+0000 7f6d51a93640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c0b28a0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.370+0000 7f6d51a93640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c0b2a30 con 0x7f6d4c1430c0 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.370+0000 7f6d51a93640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:18.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.370+0000 7f6d51a93640 10 monclient(hunting): authenticate will time out at 7589.659169s 2024-03-25T18:29:18.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.381+0000 7f6d4b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6d4c0a98f0 0x7f6d4c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:52828/0 (socket says 172.21.15.102:52828) 2024-03-25T18:29:18.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.381+0000 7f6d4b7fe640 1 -- 172.21.15.102:0/3699779746 learned_addr learned my addr 172.21.15.102:0/3699779746 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:18.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.390+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872324302 0 0) 0x7f6d4c0b28a0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.390+0000 7f6d4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:18.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.390+0000 7f6d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:18.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.390+0000 7f6d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:18.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.390+0000 7f6d4a7fc640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:29:18.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.390+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d34003520 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.399+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979830114 0 0) 0x7f6d4c0b2a30 con 0x7f6d4c1430c0 2024-03-25T18:29:18.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.399+0000 7f6d4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:18.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.399+0000 7f6d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:18.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.399+0000 7f6d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:18.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.399+0000 7f6d4a7fc640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:29:18.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.399+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d4c0b28a0 con 0x7f6d4c1430c0 2024-03-25T18:29:18.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063670165 0 0) 0x7f6d34003520 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 >> v1:172.21.15.102:6790/0 conn(0x7f6d4c1430c0 legacy=0x7f6d4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 >> v1:172.21.15.102:6789/0 conn(0x7f6d4c13f710 legacy=0x7f6d4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:18.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:29:18.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c0b44c0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d3c001690 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d4a7fc640 10 monclient: dump: 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:18.405 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:18.406 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:18.406 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:18.406 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:18.406 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:18.406 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d51a93640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:29:18.406 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.402+0000 7f6d51a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:18.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d3c0017f0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d4a7fc640 1 -- 172.21.15.102:0/3699779746 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d3c001c00 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d51a93640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d51a93640 4 set_mon_vals no callback set 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d4a7fc640 10 monclient: dump: 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:18.416 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:18.417 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:18.417 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:18.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d51a93640 10 monclient: shutdown 2024-03-25T18:29:18.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d51a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:18.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d51a93640 1 -- 172.21.15.102:0/3699779746 >> v1:172.21.15.146:6789/0 conn(0x7f6d4c0a98f0 legacy=0x7f6d4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.414+0000 7f6d51a93640 1 -- 172.21.15.102:0/3699779746 shutdown_connections 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.415+0000 7f6d51a93640 1 -- 172.21.15.102:0/3699779746 >> 172.21.15.102:0/3699779746 conn(0x7f6d4c0b7ea0 msgr2=0x7f6d4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.415+0000 7f6d51a93640 1 -- 172.21.15.102:0/3699779746 shutdown_connections 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.415+0000 7f6d51a93640 1 -- 172.21.15.102:0/3699779746 wait complete. 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.415+0000 7f6d51a93640 10 monclient: build_initial_monmap 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.415+0000 7f6d51a93640 10 monclient: monmap: 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:29:18.420 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.416+0000 7f6d51a93640 1 librados: starting msgr at 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.416+0000 7f6d51a93640 1 librados: starting objecter 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.416+0000 7f6d51a93640 1 Processor -- start 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.416+0000 7f6d51a93640 1 -- start start 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.416+0000 7f6d51a93640 1 librados: setting wanted keys 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.416+0000 7f6d51a93640 1 librados: calling monclient init 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.416+0000 7f6d51a93640 10 monclient: init 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 10 monclient(hunting): picked mon.noname-b con 0x7f6d4c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:18.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:18.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 10 monclient(hunting): picked mon.noname-c con 0x7f6d4c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:18.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c13c9b0 con 0x7f6d4c13f710 2024-03-25T18:29:18.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c1397c0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4c139950 con 0x7f6d4c1430c0 2024-03-25T18:29:18.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:18.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.417+0000 7f6d51a93640 10 monclient(hunting): authenticate will time out at 7589.706467s 2024-03-25T18:29:18.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.427+0000 7f6d4b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6d4c0a98f0 0x7f6d4c130f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:52858/0 (socket says 172.21.15.102:52858) 2024-03-25T18:29:18.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.427+0000 7f6d4b7fe640 1 -- 172.21.15.102:0/9807368 learned_addr learned my addr 172.21.15.102:0/9807368 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:18.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.431+0000 7f6d33fff640 10 client.?.objecter ms_handle_connect 0x7f6d4c0a98f0 2024-03-25T18:29:18.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.431+0000 7f6d33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:18.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.441+0000 7f6d33fff640 10 client.?.objecter ms_handle_connect 0x7f6d4c13f710 2024-03-25T18:29:18.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.441+0000 7f6d33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:18.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.444+0000 7f6d33fff640 10 client.?.objecter ms_handle_connect 0x7f6d4c1430c0 2024-03-25T18:29:18.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.444+0000 7f6d33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:18.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.451+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232181294 0 0) 0x7f6d4c1397c0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.451+0000 7f6d33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:18.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.451+0000 7f6d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:18.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.451+0000 7f6d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:18.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.451+0000 7f6d33fff640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:29:18.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.451+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d38004eb0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.453+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068280340 0 0) 0x7f6d4c139950 con 0x7f6d4c1430c0 2024-03-25T18:29:18.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.453+0000 7f6d33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:18.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.453+0000 7f6d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:18.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.453+0000 7f6d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:18.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.453+0000 7f6d33fff640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:29:18.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.453+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d4c1397c0 con 0x7f6d4c1430c0 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 532148949 0 0) 0x7f6d4c1397c0 con 0x7f6d4c1430c0 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d4c139950 con 0x7f6d4c1430c0 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d44001b90 con 0x7f6d4c1430c0 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 10 monclient(hunting): dump: 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.463 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652045378 0 0) 0x7f6d4c13c9b0 con 0x7f6d4c13f710 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:29:18.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.461+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d4c1397c0 con 0x7f6d4c13f710 2024-03-25T18:29:18.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.465+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4026876690 0 0) 0x7f6d38004eb0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.465+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d4c13c9b0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.465+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d3c001bc0 con 0x7f6d4c0a98f0 2024-03-25T18:29:18.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.465+0000 7f6d33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:18.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.465+0000 7f6d33fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:18.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.465+0000 7f6d33fff640 10 monclient(hunting): dump: 2024-03-25T18:29:18.467 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:18.467 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:18.468 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.468 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.468 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:18.468 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:18.468 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:18.468 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:18.468 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:18.468 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:18.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780424487 0 0) 0x7f6d4c139950 con 0x7f6d4c1430c0 2024-03-25T18:29:18.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 >> v1:172.21.15.146:6789/0 conn(0x7f6d4c0a98f0 legacy=0x7f6d4c130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 >> v1:172.21.15.102:6789/0 conn(0x7f6d4c13f710 legacy=0x7f6d4c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 1 monclient: found mon.c 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c131680 con 0x7f6d4c1430c0 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 5 monclient: authenticate success, global_id 4367 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 10 monclient: _renew_subs 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c131950 con 0x7f6d4c1430c0 2024-03-25T18:29:18.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:18.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 10 monclient: _renew_subs 2024-03-25T18:29:18.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:18.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c135a30 con 0x7f6d4c1430c0 2024-03-25T18:29:18.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.472+0000 7f6d51a93640 1 librados: init done 2024-03-25T18:29:18.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.473+0000 7f6d51a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:18.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.473+0000 7f6d51a93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:18.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.473+0000 7f6d51a93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:18.477 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.473+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c13c810 con 0x7f6d4c1430c0 2024-03-25T18:29:18.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.690+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d44002040 con 0x7f6d4c1430c0 2024-03-25T18:29:18.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.690+0000 7f6d33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:18.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.690+0000 7f6d49ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.769+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d44003990 con 0x7f6d4c1430c0 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.769+0000 7f6d33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.769+0000 7f6d33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.769+0000 7f6d33fff640 10 monclient: dump: 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:18.771 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:18.772 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:18.772 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:18.772 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:18.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.770+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f6d440199e0 con 0x7f6d4c1430c0 2024-03-25T18:29:18.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.770+0000 7f6d4b7fe640 10 monclient: get_auth_request con 0x7f6d3806dfb0 auth_method 0 2024-03-25T18:29:18.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.771+0000 7f6d33fff640 10 client.4367.objecter ms_handle_connect 0x7f6d3806dfb0 2024-03-25T18:29:18.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.774+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f6d44053720 con 0x7f6d4c1430c0 2024-03-25T18:29:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.774+0000 7f6d33fff640 10 client.4367.objecter ms_dispatch 0x7f6d4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.774+0000 7f6d33fff640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.775+0000 7f6d33fff640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.775+0000 7f6d33fff640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-25T18:29:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.775+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d4404f060 con 0x7f6d4c1430c0 2024-03-25T18:29:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.775+0000 7f6d33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:18.775+0000 7f6d33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:19.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.018+0000 7f6d51a93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:19.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.018+0000 7f6d51a93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:19.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.018+0000 7f6d51a93640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:19.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.018+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d4c13cb40 con 0x7f6d4c1430c0 2024-03-25T18:29:19.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.037+0000 7f6d33fff640 1 -- 172.21.15.102:0/9807368 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f6d44051160 con 0x7f6d4c1430c0 2024-03-25T18:29:19.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.037+0000 7f6d33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:19.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.037+0000 7f6d33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:19.039 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:29:19.039 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 10 librados: watch_flush enter 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 10 librados: watch_flush exit 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 10 client.4367.objecter successfully canceled tick 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 >> v1:172.21.15.146:6800/3848252714 conn(0x7f6d3806dfb0 legacy=0x7f6d38070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 10 monclient: shutdown 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 >> v1:172.21.15.102:6790/0 conn(0x7f6d4c1430c0 legacy=0x7f6d4c1e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 shutdown_connections 2024-03-25T18:29:19.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 >> 172.21.15.102:0/9807368 conn(0x7f6d4c0b7ea0 msgr2=0x7f6d4c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:19.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 shutdown_connections 2024-03-25T18:29:19.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 1 -- 172.21.15.102:0/9807368 wait complete. 2024-03-25T18:29:19.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:19.040+0000 7f6d51a93640 1 librados: shutdown 2024-03-25T18:29:19.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:23.411 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:39.647 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:23.416 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:39.652 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:23.450 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:39.685 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:23.468 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:39.704 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:23.480 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:39.716 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:23.485 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:39.699 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:23.488 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:39.701 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:23.512 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:39.748 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:23.522 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:39.736 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:25.858 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:25.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.972+0000 7f72520d9640 10 monclient: get_monmap_and_config 2024-03-25T18:29:25.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.972+0000 7f72520d9640 10 monclient: build_initial_monmap 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.972+0000 7f72520d9640 10 monclient: monmap: 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:29:25.973602+0000 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:29:25.973602+0000 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:29:25.974 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.973+0000 7f72520d9640 1 Processor -- start 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.973+0000 7f72520d9640 1 -- start start 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.973+0000 7f72520d9640 10 monclient: init 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 10 monclient(hunting): picked mon.noname-b con 0x7f724c0b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 10 monclient(hunting): picked mon.noname-a con 0x7f724c0b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 10 monclient(hunting): picked mon.noname-c con 0x7f724c14be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c0af940 con 0x7f724c0b1730 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c0b0a70 con 0x7f724c14be90 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c0b0d50 con 0x7f724c0b4d60 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:25.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.974+0000 7f72520d9640 10 monclient(hunting): authenticate will time out at 7597.262835s 2024-03-25T18:29:25.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.978+0000 7f724affd640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f724c0b1730 0x7f724c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42954/0 (socket says 172.21.15.102:42954) 2024-03-25T18:29:25.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.978+0000 7f724affd640 1 -- 172.21.15.102:0/1955620500 learned_addr learned my addr 172.21.15.102:0/1955620500 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.984+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207530947 0 0) 0x7f724c0af940 con 0x7f724c0b1730 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.984+0000 7f724a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.984+0000 7f724a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.984+0000 7f724a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:25.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.984+0000 7f724a7fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:29:25.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.984+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7234003520 con 0x7f724c0b1730 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 707356827 0 0) 0x7f7234003520 con 0x7f724c0b1730 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 >> v1:172.21.15.102:6790/0 conn(0x7f724c0b4d60 legacy=0x7f724c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 >> v1:172.21.15.146:6789/0 conn(0x7f724c14be90 legacy=0x7f724c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c0b1030 con 0x7f724c0b1730 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:25.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7240001690 con 0x7f724c0b1730 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f72520d9640 5 monclient: authenticate success, global_id 4383 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f72520d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f724a7fc640 10 monclient: dump: 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:25.995 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:25.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:25.993+0000 7f72520d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:26.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.011+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7240001860 con 0x7f724c0b1730 2024-03-25T18:29:26.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.011+0000 7f724a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:26.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.011+0000 7f724a7fc640 1 -- 172.21.15.102:0/1955620500 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7240001c70 con 0x7f724c0b1730 2024-03-25T18:29:26.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.011+0000 7f72520d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:26.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.011+0000 7f72520d9640 4 set_mon_vals no callback set 2024-03-25T18:29:26.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.011+0000 7f724a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:26.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.011+0000 7f724a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:26.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.011+0000 7f724a7fc640 10 monclient: dump: 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:26.013 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.012+0000 7f72520d9640 10 monclient: shutdown 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.012+0000 7f72520d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.012+0000 7f72520d9640 1 -- 172.21.15.102:0/1955620500 >> v1:172.21.15.102:6789/0 conn(0x7f724c0b1730 legacy=0x7f724c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.012+0000 7f72520d9640 1 -- 172.21.15.102:0/1955620500 shutdown_connections 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.012+0000 7f72520d9640 1 -- 172.21.15.102:0/1955620500 >> 172.21.15.102:0/1955620500 conn(0x7f724c0617a0 msgr2=0x7f724c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.012+0000 7f72520d9640 1 -- 172.21.15.102:0/1955620500 shutdown_connections 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.012+0000 7f72520d9640 1 -- 172.21.15.102:0/1955620500 wait complete. 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 10 monclient: build_initial_monmap 2024-03-25T18:29:26.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 10 monclient: monmap: 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 1 librados: starting msgr at 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 1 librados: starting objecter 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 1 Processor -- start 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 1 -- start start 2024-03-25T18:29:26.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 1 librados: setting wanted keys 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 1 librados: calling monclient init 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.013+0000 7f72520d9640 10 monclient: init 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 10 monclient(hunting): picked mon.noname-b con 0x7f724c0b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 10 monclient(hunting): picked mon.noname-c con 0x7f724c0b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 10 monclient(hunting): picked mon.noname-a con 0x7f724c14be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c13ab60 con 0x7f724c14be90 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c141c00 con 0x7f724c0b1730 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724c13adb0 con 0x7f724c0b4d60 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:26.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.014+0000 7f72520d9640 10 monclient(hunting): authenticate will time out at 7597.302580s 2024-03-25T18:29:26.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.025+0000 7f724affd640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f724c0b4d60 0x7f724c13b9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57106/0 (socket says 172.21.15.102:57106) 2024-03-25T18:29:26.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.026+0000 7f724affd640 1 -- 172.21.15.102:0/988014373 learned_addr learned my addr 172.21.15.102:0/988014373 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:26.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.027+0000 7f7233fff640 10 client.?.objecter ms_handle_connect 0x7f724c0b4d60 2024-03-25T18:29:26.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.027+0000 7f7233fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:26.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.028+0000 7f7233fff640 10 client.?.objecter ms_handle_connect 0x7f724c0b1730 2024-03-25T18:29:26.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.028+0000 7f7233fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:26.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.034+0000 7f7233fff640 10 client.?.objecter ms_handle_connect 0x7f724c14be90 2024-03-25T18:29:26.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.034+0000 7f7233fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:26.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.041+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531734841 0 0) 0x7f724c13adb0 con 0x7f724c0b4d60 2024-03-25T18:29:26.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.041+0000 7f7233fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:26.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.041+0000 7f7233fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:26.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.041+0000 7f7233fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:26.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.041+0000 7f7233fff640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:29:26.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.041+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7238004eb0 con 0x7f724c0b4d60 2024-03-25T18:29:26.112 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.111+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892158148 0 0) 0x7f7238004eb0 con 0x7f724c0b4d60 2024-03-25T18:29:26.112 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.111+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f724c13adb0 con 0x7f724c0b4d60 2024-03-25T18:29:26.112 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.111+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7240001b70 con 0x7f724c0b4d60 2024-03-25T18:29:26.112 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.111+0000 7f7233fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:26.112 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.111+0000 7f7233fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:26.112 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.111+0000 7f7233fff640 10 monclient(hunting): dump: 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:26.113 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.111+0000 7f7233fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.122+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3392847996 0 0) 0x7f724c13adb0 con 0x7f724c0b4d60 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 >> v1:172.21.15.146:6789/0 conn(0x7f724c0b1730 legacy=0x7f724c1db460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 >> v1:172.21.15.102:6789/0 conn(0x7f724c14be90 legacy=0x7f724c1e30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 1 monclient: found mon.c 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c1daf40 con 0x7f724c0b4d60 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f7233fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:26.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:29:26.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 10 monclient: _renew_subs 2024-03-25T18:29:26.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:26.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f724c1dbbd0 con 0x7f724c0b4d60 2024-03-25T18:29:26.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:26.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 10 monclient: _renew_subs 2024-03-25T18:29:26.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:26.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f724c13c330 con 0x7f724c0b4d60 2024-03-25T18:29:26.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.123+0000 7f72520d9640 1 librados: init done 2024-03-25T18:29:26.126 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.124+0000 7f72520d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:26.126 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.124+0000 7f72520d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:26.126 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.124+0000 7f72520d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:26.126 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.124+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f724c0627b0 con 0x7f724c0b4d60 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.426+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7240004bf0 con 0x7f724c0b4d60 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.426+0000 7f7233fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.426+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7240004f00 con 0x7f724c0b4d60 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.426+0000 7f7233fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.426+0000 7f7233fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.426+0000 7f7233fff640 10 monclient: dump: 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:26.427 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:26.428 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:26.428 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:26.428 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:26.428 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:26.428 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:26.428 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:26.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.426+0000 7f7249ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:26.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.427+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f724001aed0 con 0x7f724c0b4d60 2024-03-25T18:29:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.427+0000 7f724b7fe640 10 monclient: get_auth_request con 0x7f723806dc40 auth_method 0 2024-03-25T18:29:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.428+0000 7f7233fff640 10 client.4370.objecter ms_handle_connect 0x7f723806dc40 2024-03-25T18:29:26.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.434+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f72400549d0 con 0x7f724c0b4d60 2024-03-25T18:29:26.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.434+0000 7f7233fff640 10 client.4370.objecter ms_dispatch 0x7f724c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:26.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.434+0000 7f7233fff640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:26.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.434+0000 7f7233fff640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:26.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.434+0000 7f7233fff640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-25T18:29:26.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.434+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72400504e0 con 0x7f724c0b4d60 2024-03-25T18:29:26.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.435+0000 7f7233fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:26.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.435+0000 7f7233fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:26.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.688+0000 7f72520d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:26.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.688+0000 7f72520d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:26.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.688+0000 7f72520d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:26.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.688+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f724c069f40 con 0x7f724c0b4d60 2024-03-25T18:29:26.725 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.724+0000 7f7233fff640 1 -- 172.21.15.102:0/988014373 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f7240052410 con 0x7f724c0b4d60 2024-03-25T18:29:26.725 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.724+0000 7f7233fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:26.725 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.724+0000 7f7233fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:26.726 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:29:26.726 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:26.729 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 10 librados: watch_flush enter 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 10 librados: watch_flush exit 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 10 client.4370.objecter successfully canceled tick 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 >> v1:172.21.15.146:6800/3848252714 conn(0x7f723806dc40 legacy=0x7f7238070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 10 monclient: shutdown 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 >> v1:172.21.15.102:6790/0 conn(0x7f724c0b4d60 legacy=0x7f724c13b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 shutdown_connections 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 >> 172.21.15.102:0/988014373 conn(0x7f724c0617a0 msgr2=0x7f724c0b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 shutdown_connections 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.727+0000 7f72520d9640 1 -- 172.21.15.102:0/988014373 wait complete. 2024-03-25T18:29:26.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:26.728+0000 7f72520d9640 1 librados: shutdown 2024-03-25T18:29:26.739 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:33.546 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:33.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.662+0000 7f9b858c8640 10 monclient: get_monmap_and_config 2024-03-25T18:29:33.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.662+0000 7f9b858c8640 10 monclient: build_initial_monmap 2024-03-25T18:29:33.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.662+0000 7f9b858c8640 10 monclient: monmap: 2024-03-25T18:29:33.663 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:33.663 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:33.663 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:29:33.663387+0000 2024-03-25T18:29:33.664 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:29:33.663387+0000 2024-03-25T18:29:33.664 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:33.664 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:33.664 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:33.664 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:29:33.664 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:29:33.664 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:33.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.663+0000 7f9b858c8640 1 Processor -- start 2024-03-25T18:29:33.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.663+0000 7f9b858c8640 1 -- start start 2024-03-25T18:29:33.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.663+0000 7f9b858c8640 10 monclient: init 2024-03-25T18:29:33.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:33.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:33.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b800a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:33.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b8013f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:33.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b801430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:33.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b80142520 con 0x7f9b801430c0 2024-03-25T18:29:33.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b800b28a0 con 0x7f9b8013f710 2024-03-25T18:29:33.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b800b2a30 con 0x7f9b800a98f0 2024-03-25T18:29:33.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:33.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.664+0000 7f9b858c8640 10 monclient(hunting): authenticate will time out at 7604.952428s 2024-03-25T18:29:33.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.689+0000 7f9b77fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9b8013f710 0x7f9b801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:55026/0 (socket says 172.21.15.102:55026) 2024-03-25T18:29:33.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.689+0000 7f9b77fff640 1 -- 172.21.15.102:0/1530690496 learned_addr learned my addr 172.21.15.102:0/1530690496 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:33.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.704+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791638462 0 0) 0x7f9b800b28a0 con 0x7f9b8013f710 2024-03-25T18:29:33.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.704+0000 7f9b7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:33.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.704+0000 7f9b7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:33.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.704+0000 7f9b7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:33.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.704+0000 7f9b7e7fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:29:33.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.704+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b6c003520 con 0x7f9b8013f710 2024-03-25T18:29:33.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.705+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160426697 0 0) 0x7f9b80142520 con 0x7f9b801430c0 2024-03-25T18:29:33.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.705+0000 7f9b7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:33.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.705+0000 7f9b7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:33.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.705+0000 7f9b7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:33.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.705+0000 7f9b7e7fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:29:33.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.705+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b800b28a0 con 0x7f9b801430c0 2024-03-25T18:29:33.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.714+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644260539 0 0) 0x7f9b800b2a30 con 0x7f9b800a98f0 2024-03-25T18:29:33.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.714+0000 7f9b7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:33.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.714+0000 7f9b7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:33.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.714+0000 7f9b7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:33.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.714+0000 7f9b7e7fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:29:33.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.714+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b80142520 con 0x7f9b800a98f0 2024-03-25T18:29:33.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850114645 0 0) 0x7f9b6c003520 con 0x7f9b8013f710 2024-03-25T18:29:33.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 >> v1:172.21.15.102:6790/0 conn(0x7f9b800a98f0 legacy=0x7f9b800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 >> v1:172.21.15.102:6789/0 conn(0x7f9b801430c0 legacy=0x7f9b8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800b44c0 con 0x7f9b8013f710 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b68001690 con 0x7f9b8013f710 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b7e7fc640 10 monclient: dump: 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:33.722 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b858c8640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:29:33.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.720+0000 7f9b858c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:33.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.739+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b680017f0 con 0x7f9b8013f710 2024-03-25T18:29:33.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.739+0000 7f9b7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:33.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.739+0000 7f9b7e7fc640 1 -- 172.21.15.102:0/1530690496 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b68001c00 con 0x7f9b8013f710 2024-03-25T18:29:33.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.739+0000 7f9b858c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:33.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.739+0000 7f9b858c8640 4 set_mon_vals no callback set 2024-03-25T18:29:33.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.739+0000 7f9b7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.739+0000 7f9b7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.739+0000 7f9b7e7fc640 10 monclient: dump: 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:33.741 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:33.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.740+0000 7f9b858c8640 10 monclient: shutdown 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.740+0000 7f9b858c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.740+0000 7f9b858c8640 1 -- 172.21.15.102:0/1530690496 >> v1:172.21.15.146:6789/0 conn(0x7f9b8013f710 legacy=0x7f9b801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.740+0000 7f9b858c8640 1 -- 172.21.15.102:0/1530690496 shutdown_connections 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.740+0000 7f9b858c8640 1 -- 172.21.15.102:0/1530690496 >> 172.21.15.102:0/1530690496 conn(0x7f9b800b7ea0 msgr2=0x7f9b800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.740+0000 7f9b858c8640 1 -- 172.21.15.102:0/1530690496 shutdown_connections 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.740+0000 7f9b858c8640 1 -- 172.21.15.102:0/1530690496 wait complete. 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 10 monclient: build_initial_monmap 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 10 monclient: monmap: 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:33.744 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 1 librados: starting msgr at 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 1 librados: starting objecter 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 1 Processor -- start 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 1 -- start start 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 1 librados: setting wanted keys 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 1 librados: calling monclient init 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.741+0000 7f9b858c8640 10 monclient: init 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 10 monclient(hunting): picked mon.noname-a con 0x7f9b800a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 10 monclient(hunting): picked mon.noname-c con 0x7f9b8013f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:33.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 10 monclient(hunting): picked mon.noname-b con 0x7f9b801430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:33.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b80139a40 con 0x7f9b800a98f0 2024-03-25T18:29:33.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b80133f20 con 0x7f9b801430c0 2024-03-25T18:29:33.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b800ab950 con 0x7f9b8013f710 2024-03-25T18:29:33.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:33.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.742+0000 7f9b858c8640 10 monclient(hunting): authenticate will time out at 7605.030508s 2024-03-25T18:29:33.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.753+0000 7f9b7f7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9b801430c0 0x7f9b80133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:55088/0 (socket says 172.21.15.102:55088) 2024-03-25T18:29:33.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.753+0000 7f9b7f7fe640 1 -- 172.21.15.102:0/2408870999 learned_addr learned my addr 172.21.15.102:0/2408870999 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:33.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.754+0000 7f9b7cff9640 10 client.?.objecter ms_handle_connect 0x7f9b801430c0 2024-03-25T18:29:33.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.754+0000 7f9b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:33.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.755+0000 7f9b7cff9640 10 client.?.objecter ms_handle_connect 0x7f9b8013f710 2024-03-25T18:29:33.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.755+0000 7f9b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:33.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.762+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932860770 0 0) 0x7f9b800ab950 con 0x7f9b8013f710 2024-03-25T18:29:33.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.762+0000 7f9b7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:33.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.762+0000 7f9b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:33.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.762+0000 7f9b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:33.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.762+0000 7f9b7cff9640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:29:33.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.762+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b60004eb0 con 0x7f9b8013f710 2024-03-25T18:29:33.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.771+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237870100 0 0) 0x7f9b80133f20 con 0x7f9b801430c0 2024-03-25T18:29:33.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.771+0000 7f9b7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:33.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.771+0000 7f9b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:33.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.771+0000 7f9b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:33.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.771+0000 7f9b7cff9640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:29:33.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.771+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b800ab950 con 0x7f9b801430c0 2024-03-25T18:29:33.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.772+0000 7f9b7cff9640 10 client.?.objecter ms_handle_connect 0x7f9b800a98f0 2024-03-25T18:29:33.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.772+0000 7f9b7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:33.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.778+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726931603 0 0) 0x7f9b80139a40 con 0x7f9b800a98f0 2024-03-25T18:29:33.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.778+0000 7f9b7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:33.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.778+0000 7f9b7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:33.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.778+0000 7f9b7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:33.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.778+0000 7f9b7cff9640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:29:33.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.778+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b80133f20 con 0x7f9b800a98f0 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.781+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 364036249 0 0) 0x7f9b60004eb0 con 0x7f9b8013f710 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.781+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b80139a40 con 0x7f9b8013f710 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.781+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b68001b00 con 0x7f9b8013f710 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.781+0000 7f9b7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.781+0000 7f9b7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.781+0000 7f9b7cff9640 10 monclient(hunting): dump: 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:33.782 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:33.783 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:33.783 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:33.783 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:33.783 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:33.783 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:33.783 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:33.783 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:33.783 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.781+0000 7f9b7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 867968451 0 0) 0x7f9b80139a40 con 0x7f9b8013f710 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 >> v1:172.21.15.146:6789/0 conn(0x7f9b801430c0 legacy=0x7f9b80133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 >> v1:172.21.15.102:6789/0 conn(0x7f9b800a98f0 legacy=0x7f9b80138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 1 monclient: found mon.c 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800abc30 con 0x7f9b8013f710 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:33.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b858c8640 5 monclient: authenticate success, global_id 4376 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b858c8640 10 monclient: _renew_subs 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b858c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.792+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b800abf10 con 0x7f9b8013f710 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.793+0000 7f9b858c8640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.793+0000 7f9b858c8640 10 monclient: _renew_subs 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.793+0000 7f9b858c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.793+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b80139370 con 0x7f9b8013f710 2024-03-25T18:29:33.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.793+0000 7f9b858c8640 1 librados: init done 2024-03-25T18:29:33.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.794+0000 7f9b858c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:33.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.794+0000 7f9b858c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:33.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.794+0000 7f9b858c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:33.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:33.794+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b800627b0 con 0x7f9b8013f710 2024-03-25T18:29:34.091 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.090+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b68004b00 con 0x7f9b8013f710 2024-03-25T18:29:34.091 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.090+0000 7f9b7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.090+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b68004ed0 con 0x7f9b8013f710 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.090+0000 7f9b7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.090+0000 7f9b7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.090+0000 7f9b7cff9640 10 monclient: dump: 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:34.092 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:34.093 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.090+0000 7f9b7dffb640 4 set_mon_vals no callback set 2024-03-25T18:29:34.093 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.091+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f9b680050f0 con 0x7f9b8013f710 2024-03-25T18:29:34.093 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.091+0000 7f9b7effd640 10 monclient: get_auth_request con 0x7f9b6006de00 auth_method 0 2024-03-25T18:29:34.093 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.092+0000 7f9b7cff9640 10 client.4376.objecter ms_handle_connect 0x7f9b6006de00 2024-03-25T18:29:34.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.099+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f9b68001c60 con 0x7f9b8013f710 2024-03-25T18:29:34.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.099+0000 7f9b7cff9640 10 client.4376.objecter ms_dispatch 0x7f9b80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:34.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.099+0000 7f9b7cff9640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:34.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.099+0000 7f9b7cff9640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:34.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.099+0000 7f9b7cff9640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T18:29:34.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.099+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b680503a0 con 0x7f9b8013f710 2024-03-25T18:29:34.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.099+0000 7f9b7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:34.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.099+0000 7f9b7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:34.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.351+0000 7f9b858c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:34.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.351+0000 7f9b858c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:34.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.351+0000 7f9b858c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:29:34.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.351+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b800b7180 con 0x7f9b8013f710 2024-03-25T18:29:34.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.380+0000 7f9b7cff9640 1 -- 172.21.15.102:0/2408870999 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f9b680522d0 con 0x7f9b8013f710 2024-03-25T18:29:34.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.380+0000 7f9b7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:34.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.380+0000 7f9b7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:34.382 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:29:34.382 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:34.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 10 librados: watch_flush enter 2024-03-25T18:29:34.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 10 librados: watch_flush exit 2024-03-25T18:29:34.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T18:29:34.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 10 client.4376.objecter successfully canceled tick 2024-03-25T18:29:34.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 >> v1:172.21.15.146:6800/3848252714 conn(0x7f9b6006de00 legacy=0x7f9b600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:34.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 10 monclient: shutdown 2024-03-25T18:29:34.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:34.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 >> v1:172.21.15.102:6790/0 conn(0x7f9b8013f710 legacy=0x7f9b80133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:34.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 shutdown_connections 2024-03-25T18:29:34.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 >> 172.21.15.102:0/2408870999 conn(0x7f9b800b7ea0 msgr2=0x7f9b80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:34.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 shutdown_connections 2024-03-25T18:29:34.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 1 -- 172.21.15.102:0/2408870999 wait complete. 2024-03-25T18:29:34.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:34.383+0000 7f9b858c8640 1 librados: shutdown 2024-03-25T18:29:34.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:41.202 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.295+0000 7ff93adcf640 10 monclient: get_monmap_and_config 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.295+0000 7ff93adcf640 10 monclient: build_initial_monmap 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.295+0000 7ff93adcf640 10 monclient: monmap: 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:29:41.296147+0000 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:29:41.296147+0000 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:41.296 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:41.297 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:41.297 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:29:41.297 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:29:41.297 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:41.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.296+0000 7ff93adcf640 1 Processor -- start 2024-03-25T18:29:41.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.296+0000 7ff93adcf640 1 -- start start 2024-03-25T18:29:41.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.296+0000 7ff93adcf640 10 monclient: init 2024-03-25T18:29:41.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:41.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:41.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 10 monclient(hunting): picked mon.noname-a con 0x7ff9340b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:41.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 10 monclient(hunting): picked mon.noname-c con 0x7ff9340b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:41.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 10 monclient(hunting): picked mon.noname-b con 0x7ff93414be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:41.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9340af940 con 0x7ff9340b4d60 2024-03-25T18:29:41.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9340b0a70 con 0x7ff9340b1730 2024-03-25T18:29:41.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9340b0d50 con 0x7ff93414be90 2024-03-25T18:29:41.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:41.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.297+0000 7ff93adcf640 10 monclient(hunting): authenticate will time out at 7612.585695s 2024-03-25T18:29:41.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.308+0000 7ff938b6b640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff9340b4d60 0x7ff93413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57768/0 (socket says 172.21.15.102:57768) 2024-03-25T18:29:41.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.308+0000 7ff938b6b640 1 -- 172.21.15.102:0/3511013912 learned_addr learned my addr 172.21.15.102:0/3511013912 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:41.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.323+0000 7ff9337fe640 1 -- 172.21.15.102:0/3511013912 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183705400 0 0) 0x7ff9340b0d50 con 0x7ff93414be90 2024-03-25T18:29:41.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.323+0000 7ff9337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:41.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.323+0000 7ff9337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:41.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.323+0000 7ff9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:41.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.323+0000 7ff9337fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:29:41.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.323+0000 7ff9337fe640 1 -- 172.21.15.102:0/3511013912 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff924003520 con 0x7ff93414be90 2024-03-25T18:29:41.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 1 -- 172.21.15.102:0/3511013912 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515221286 0 0) 0x7ff924003520 con 0x7ff93414be90 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 1 -- 172.21.15.102:0/3511013912 >> v1:172.21.15.146:6789/0 conn(0x7ff9340b1730 legacy=0x7ff9340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 1 -- 172.21.15.102:0/3511013912 >> v1:172.21.15.102:6789/0 conn(0x7ff9340b4d60 legacy=0x7ff93413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 1 monclient: found mon.noname-b 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 1 -- 172.21.15.102:0/3511013912 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9340b1030 con 0x7ff93414be90 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:41.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 1 -- 172.21.15.102:0/3511013912 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff92c001a10 con 0x7ff93414be90 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff93adcf640 5 monclient: authenticate success, global_id 4379 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff93adcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff9337fe640 10 monclient: dump: 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:41.333 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:41.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.330+0000 7ff93adcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:41.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff9337fe640 1 -- 172.21.15.102:0/3511013912 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff92c001b70 con 0x7ff93414be90 2024-03-25T18:29:41.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff9337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:41.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:41.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 4 set_mon_vals no callback set 2024-03-25T18:29:41.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 10 monclient: shutdown 2024-03-25T18:29:41.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:41.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 1 -- 172.21.15.102:0/3511013912 >> v1:172.21.15.102:6790/0 conn(0x7ff93414be90 legacy=0x7ff93414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 1 -- 172.21.15.102:0/3511013912 shutdown_connections 2024-03-25T18:29:41.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 1 -- 172.21.15.102:0/3511013912 >> 172.21.15.102:0/3511013912 conn(0x7ff9340617a0 msgr2=0x7ff934061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:41.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 1 -- 172.21.15.102:0/3511013912 shutdown_connections 2024-03-25T18:29:41.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.336+0000 7ff93adcf640 1 -- 172.21.15.102:0/3511013912 wait complete. 2024-03-25T18:29:41.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.337+0000 7ff93adcf640 10 monclient: build_initial_monmap 2024-03-25T18:29:41.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.337+0000 7ff93adcf640 10 monclient: monmap: 2024-03-25T18:29:41.339 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:41.339 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:41.339 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.337+0000 7ff93adcf640 1 librados: starting msgr at 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.337+0000 7ff93adcf640 1 librados: starting objecter 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 1 Processor -- start 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 1 -- start start 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 1 librados: setting wanted keys 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 1 librados: calling monclient init 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 10 monclient: init 2024-03-25T18:29:41.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 10 monclient(hunting): picked mon.noname-b con 0x7ff9340b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 10 monclient(hunting): picked mon.noname-c con 0x7ff9340b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 10 monclient(hunting): picked mon.noname-a con 0x7ff93414be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff934140860 con 0x7ff93414be90 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff934137d70 con 0x7ff9340b1730 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff934140380 con 0x7ff9340b4d60 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:41.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.338+0000 7ff93adcf640 10 monclient(hunting): authenticate will time out at 7612.626914s 2024-03-25T18:29:41.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.343+0000 7ff933fff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7ff9340b4d60 0x7ff9341d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:50422/0 (socket says 172.21.15.102:50422) 2024-03-25T18:29:41.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.343+0000 7ff933fff640 1 -- 172.21.15.102:0/1351904367 learned_addr learned my addr 172.21.15.102:0/1351904367 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:41.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.346+0000 7ff9317fa640 10 client.?.objecter ms_handle_connect 0x7ff9340b4d60 2024-03-25T18:29:41.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.346+0000 7ff9317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:41.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.349+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220157694 0 0) 0x7ff934140380 con 0x7ff9340b4d60 2024-03-25T18:29:41.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.349+0000 7ff9317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:41.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.349+0000 7ff9317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:41.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.349+0000 7ff9317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:41.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.349+0000 7ff9317fa640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:29:41.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.349+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff918004eb0 con 0x7ff9340b4d60 2024-03-25T18:29:41.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.350+0000 7ff9317fa640 10 client.?.objecter ms_handle_connect 0x7ff93414be90 2024-03-25T18:29:41.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.350+0000 7ff9317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:41.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.352+0000 7ff9317fa640 10 client.?.objecter ms_handle_connect 0x7ff9340b1730 2024-03-25T18:29:41.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.352+0000 7ff9317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:41.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.359+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429939375 0 0) 0x7ff934140860 con 0x7ff93414be90 2024-03-25T18:29:41.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.359+0000 7ff9317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:41.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.359+0000 7ff9317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:41.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.359+0000 7ff9317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:41.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.359+0000 7ff9317fa640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:29:41.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.359+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff934140380 con 0x7ff93414be90 2024-03-25T18:29:41.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.366+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008169099 0 0) 0x7ff934137d70 con 0x7ff9340b1730 2024-03-25T18:29:41.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.366+0000 7ff9317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:41.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.366+0000 7ff9317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:41.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.366+0000 7ff9317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:41.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.366+0000 7ff9317fa640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:29:41.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.366+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff934140860 con 0x7ff9340b1730 2024-03-25T18:29:41.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.367+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 477087589 0 0) 0x7ff934140380 con 0x7ff93414be90 2024-03-25T18:29:41.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.367+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff934137d70 con 0x7ff93414be90 2024-03-25T18:29:41.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.367+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff92c001e80 con 0x7ff93414be90 2024-03-25T18:29:41.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.367+0000 7ff9317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:41.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.367+0000 7ff9317fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:41.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.367+0000 7ff9317fa640 10 monclient(hunting): dump: 2024-03-25T18:29:41.368 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:41.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.367+0000 7ff9317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.378+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3235445975 0 0) 0x7ff918004eb0 con 0x7ff9340b4d60 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.378+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff934140380 con 0x7ff9340b4d60 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.378+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff920001a30 con 0x7ff9340b4d60 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.378+0000 7ff9317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.378+0000 7ff9317fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.378+0000 7ff9317fa640 10 monclient(hunting): dump: 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:41.379 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:41.380 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:41.380 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:41.380 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:41.380 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:41.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 260518463 0 0) 0x7ff934137d70 con 0x7ff93414be90 2024-03-25T18:29:41.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 >> v1:172.21.15.102:6790/0 conn(0x7ff9340b4d60 legacy=0x7ff9341d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 >> v1:172.21.15.146:6789/0 conn(0x7ff9340b1730 legacy=0x7ff934131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 1 monclient: found mon.a 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff934140510 con 0x7ff93414be90 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 10 monclient: _finish_auth 0 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff9317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 5 monclient: authenticate success, global_id 4398 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 10 monclient: _renew_subs 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff934132520 con 0x7ff93414be90 2024-03-25T18:29:41.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:41.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 10 monclient: _renew_subs 2024-03-25T18:29:41.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:41.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff934138230 con 0x7ff93414be90 2024-03-25T18:29:41.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.382+0000 7ff93adcf640 1 librados: init done 2024-03-25T18:29:41.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.383+0000 7ff93adcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:41.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.383+0000 7ff93adcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:41.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.383+0000 7ff93adcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:41.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.383+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff934132820 con 0x7ff93414be90 2024-03-25T18:29:41.679 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.678+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff92c0020e0 con 0x7ff93414be90 2024-03-25T18:29:41.679 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.678+0000 7ff9317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:41.679 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.678+0000 7ff932ffd640 4 set_mon_vals no callback set 2024-03-25T18:29:41.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.688+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ff92c0015d0 con 0x7ff93414be90 2024-03-25T18:29:41.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.688+0000 7ff9317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:41.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.688+0000 7ff9317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:41.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.688+0000 7ff9317fa640 10 monclient: dump: 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:41.690 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:41.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.689+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7ff92c019cf0 con 0x7ff93414be90 2024-03-25T18:29:41.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.690+0000 7ff938b6b640 10 monclient: get_auth_request con 0x7ff91806df10 auth_method 0 2024-03-25T18:29:41.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.690+0000 7ff9317fa640 10 client.4398.objecter ms_handle_connect 0x7ff91806df10 2024-03-25T18:29:41.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.695+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7ff92c053980 con 0x7ff93414be90 2024-03-25T18:29:41.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.695+0000 7ff9317fa640 10 client.4398.objecter ms_dispatch 0x7ff934000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:41.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.695+0000 7ff9317fa640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:41.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.695+0000 7ff9317fa640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:41.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.695+0000 7ff9317fa640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-25T18:29:41.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.695+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff92c04f2c0 con 0x7ff93414be90 2024-03-25T18:29:41.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.695+0000 7ff9317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:41.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.695+0000 7ff9317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:41.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.946+0000 7ff93adcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:41.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.947+0000 7ff93adcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:41.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.947+0000 7ff93adcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:41.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.947+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9341409f0 con 0x7ff93414be90 2024-03-25T18:29:41.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.979+0000 7ff9317fa640 1 -- 172.21.15.102:0/1351904367 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7ff92c0513c0 con 0x7ff93414be90 2024-03-25T18:29:41.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.979+0000 7ff9317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:41.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.979+0000 7ff9317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:41.980 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:29:41.980 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:41.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 10 librados: watch_flush enter 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 10 librados: watch_flush exit 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 10 client.4398.objecter successfully canceled tick 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 >> v1:172.21.15.146:6800/3848252714 conn(0x7ff91806df10 legacy=0x7ff9180703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 10 monclient: shutdown 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 >> v1:172.21.15.102:6789/0 conn(0x7ff93414be90 legacy=0x7ff9341d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.981+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 shutdown_connections 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.982+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 >> 172.21.15.102:0/1351904367 conn(0x7ff9340617a0 msgr2=0x7ff9340b68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.982+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 shutdown_connections 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.982+0000 7ff93adcf640 1 -- 172.21.15.102:0/1351904367 wait complete. 2024-03-25T18:29:41.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:41.982+0000 7ff93adcf640 1 librados: shutdown 2024-03-25T18:29:41.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:43.439 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:59.674 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:43.496 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:59.732 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:43.529 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:03:59.764 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:43.531 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:59.744 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:43.543 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:59.756 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:43.552 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:59.788 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:43.558 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:59.794 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:43.592 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:03:59.805 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:43.606 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:03:59.842 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:48.801 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:48.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.914+0000 7fbbc7560640 10 monclient: get_monmap_and_config 2024-03-25T18:29:48.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.914+0000 7fbbc7560640 10 monclient: build_initial_monmap 2024-03-25T18:29:48.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.914+0000 7fbbc7560640 10 monclient: monmap: 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:29:48.915543+0000 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:29:48.915543+0000 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:29:48.916 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.915+0000 7fbbc7560640 1 Processor -- start 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.915+0000 7fbbc7560640 1 -- start start 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.915+0000 7fbbc7560640 10 monclient: init 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc00b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 10 monclient(hunting): picked mon.noname-a con 0x7fbbc00b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 10 monclient(hunting): picked mon.noname-b con 0x7fbbc014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:48.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00af940 con 0x7fbbc00b1730 2024-03-25T18:29:48.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00b0a70 con 0x7fbbc00b4d60 2024-03-25T18:29:48.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00b0d50 con 0x7fbbc014be90 2024-03-25T18:29:48.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:48.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.916+0000 7fbbc7560640 10 monclient(hunting): authenticate will time out at 7620.204418s 2024-03-25T18:29:48.932 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.931+0000 7fbbc5afd640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fbbc014be90 0x7fbbc014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:49612/0 (socket says 172.21.15.102:49612) 2024-03-25T18:29:48.932 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.931+0000 7fbbc5afd640 1 -- 172.21.15.102:0/4066703358 learned_addr learned my addr 172.21.15.102:0/4066703358 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:48.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.945+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691180825 0 0) 0x7fbbc00b0a70 con 0x7fbbc00b4d60 2024-03-25T18:29:48.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.945+0000 7fbbb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:48.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.945+0000 7fbbb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:48.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.945+0000 7fbbb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:48.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.945+0000 7fbbb7fff640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:29:48.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.945+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbac003520 con 0x7fbbc00b4d60 2024-03-25T18:29:48.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.949+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800626922 0 0) 0x7fbbc00b0d50 con 0x7fbbc014be90 2024-03-25T18:29:48.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.949+0000 7fbbb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:48.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.949+0000 7fbbb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:48.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.949+0000 7fbbb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:48.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.949+0000 7fbbb7fff640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:29:48.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.949+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbc00b0a70 con 0x7fbbc014be90 2024-03-25T18:29:48.961 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3815253802 0 0) 0x7fbbc00b0a70 con 0x7fbbc014be90 2024-03-25T18:29:48.961 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 >> v1:172.21.15.146:6789/0 conn(0x7fbbc00b4d60 legacy=0x7fbbc013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 >> v1:172.21.15.102:6789/0 conn(0x7fbbc00b1730 legacy=0x7fbbc00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 1 monclient: found mon.noname-b 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc00b1030 con 0x7fbbc014be90 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 10 monclient: _finish_auth 0 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbb80019f0 con 0x7fbbc014be90 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:48.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbb7fff640 10 monclient: dump: 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbc7560640 5 monclient: authenticate success, global_id 4385 2024-03-25T18:29:48.963 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.961+0000 7fbbc7560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:48.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.974+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbb8001b50 con 0x7fbbc014be90 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.974+0000 7fbbb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.974+0000 7fbbb7fff640 1 -- 172.21.15.102:0/4066703358 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbb8001f60 con 0x7fbbc014be90 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.974+0000 7fbbc7560640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.974+0000 7fbbc7560640 4 set_mon_vals no callback set 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.974+0000 7fbbb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.974+0000 7fbbb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.974+0000 7fbbb7fff640 10 monclient: dump: 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:48.975 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:48.976 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:48.976 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:48.976 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:48.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.975+0000 7fbbc7560640 10 monclient: shutdown 2024-03-25T18:29:48.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.975+0000 7fbbc7560640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:48.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.975+0000 7fbbc7560640 1 -- 172.21.15.102:0/4066703358 >> v1:172.21.15.102:6790/0 conn(0x7fbbc014be90 legacy=0x7fbbc014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:48.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.975+0000 7fbbc7560640 1 -- 172.21.15.102:0/4066703358 shutdown_connections 2024-03-25T18:29:48.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.975+0000 7fbbc7560640 1 -- 172.21.15.102:0/4066703358 >> 172.21.15.102:0/4066703358 conn(0x7fbbc00617a0 msgr2=0x7fbbc0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:48.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.975+0000 7fbbc7560640 1 -- 172.21.15.102:0/4066703358 shutdown_connections 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.975+0000 7fbbc7560640 1 -- 172.21.15.102:0/4066703358 wait complete. 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 10 monclient: build_initial_monmap 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 10 monclient: monmap: 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:29:48.978 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:48.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 1 librados: starting msgr at 2024-03-25T18:29:48.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 1 librados: starting objecter 2024-03-25T18:29:48.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 1 Processor -- start 2024-03-25T18:29:48.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 1 -- start start 2024-03-25T18:29:48.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 1 librados: setting wanted keys 2024-03-25T18:29:48.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 1 librados: calling monclient init 2024-03-25T18:29:48.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.976+0000 7fbbc7560640 10 monclient: init 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc00b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 10 monclient(hunting): picked mon.noname-a con 0x7fbbc00b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 10 monclient(hunting): picked mon.noname-b con 0x7fbbc014be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00a9aa0 con 0x7fbbc00b4d60 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc0141c50 con 0x7fbbc014be90 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc0142880 con 0x7fbbc00b1730 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:48.980 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.977+0000 7fbbc7560640 10 monclient(hunting): authenticate will time out at 7620.265438s 2024-03-25T18:29:48.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.993+0000 7fbbc4afb640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbbc00b4d60 0x7fbbc0137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33148/0 (socket says 172.21.15.102:33148) 2024-03-25T18:29:48.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.993+0000 7fbbc4afb640 1 -- 172.21.15.102:0/7577185 learned_addr learned my addr 172.21.15.102:0/7577185 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:48.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.993+0000 7fbbb5ffb640 10 client.?.objecter ms_handle_connect 0x7fbbc00b4d60 2024-03-25T18:29:48.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.993+0000 7fbbb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:48.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.994+0000 7fbbb5ffb640 10 client.?.objecter ms_handle_connect 0x7fbbc00b1730 2024-03-25T18:29:48.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:48.994+0000 7fbbb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:49.001 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.001+0000 7fbbb5ffb640 10 client.?.objecter ms_handle_connect 0x7fbbc014be90 2024-03-25T18:29:49.001 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.001+0000 7fbbb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:49.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.005+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100350247 0 0) 0x7fbbc00a9aa0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.005+0000 7fbbb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:49.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.005+0000 7fbbb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:49.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.005+0000 7fbbb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:49.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.005+0000 7fbbb5ffb640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:29:49.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.005+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba4004eb0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.012+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2801870061 0 0) 0x7fbba4004eb0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.012+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbc00a9aa0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.012+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbb00019d0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.012+0000 7fbbb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:49.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.012+0000 7fbbb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:49.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.012+0000 7fbbb5ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:49.013 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:49.013 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:49.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.012+0000 7fbbb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230148878 0 0) 0x7fbbc0141c50 con 0x7fbbc014be90 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba4004eb0 con 0x7fbbc014be90 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709739949 0 0) 0x7fbbc0142880 con 0x7fbbc00b1730 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:29:49.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.016+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbc0141c50 con 0x7fbbc00b1730 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 715133729 0 0) 0x7fbbc00a9aa0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 >> v1:172.21.15.102:6790/0 conn(0x7fbbc00b1730 legacy=0x7fbbc0142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 >> v1:172.21.15.146:6789/0 conn(0x7fbbc014be90 legacy=0x7fbbc01e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 1 monclient: found mon.a 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc0142a10 con 0x7fbbc00b4d60 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:49.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 10 monclient: _renew_subs 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc01db730 con 0x7fbbc00b4d60 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 10 monclient: _renew_subs 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc013a690 con 0x7fbbc00b4d60 2024-03-25T18:29:49.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.028+0000 7fbbc7560640 1 librados: init done 2024-03-25T18:29:49.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.029+0000 7fbbc7560640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:49.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.029+0000 7fbbc7560640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:49.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.029+0000 7fbbc7560640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:49.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.029+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc0141a40 con 0x7fbbc00b4d60 2024-03-25T18:29:49.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.328+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbb0001c50 con 0x7fbbc00b4d60 2024-03-25T18:29:49.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.328+0000 7fbbb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:49.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.328+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fbbb00038c0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.328+0000 7fbbb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:49.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.328+0000 7fbbb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:49.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.328+0000 7fbbb5ffb640 10 monclient: dump: 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.328+0000 7fbbb77fe640 4 set_mon_vals no callback set 2024-03-25T18:29:49.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.329+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fbbb0003ae0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.329+0000 7fbbc52fc640 10 monclient: get_auth_request con 0x7fbba406de10 auth_method 0 2024-03-25T18:29:49.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.330+0000 7fbbb5ffb640 10 client.4404.objecter ms_handle_connect 0x7fbba406de10 2024-03-25T18:29:49.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.334+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fbbb00546f0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.334+0000 7fbbb5ffb640 10 client.4404.objecter ms_dispatch 0x7fbbc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:49.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.334+0000 7fbbb5ffb640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:49.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.334+0000 7fbbb5ffb640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:49.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.334+0000 7fbbb5ffb640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T18:29:49.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.334+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbb0050200 con 0x7fbbc00b4d60 2024-03-25T18:29:49.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.334+0000 7fbbb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:49.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.334+0000 7fbbb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:49.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.568+0000 7fbbc7560640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:49.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.568+0000 7fbbc7560640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:49.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.568+0000 7fbbc7560640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:49.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.568+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbc0141de0 con 0x7fbbc00b4d60 2024-03-25T18:29:49.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.594+0000 7fbbb5ffb640 1 -- 172.21.15.102:0/7577185 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fbbb0052130 con 0x7fbbc00b4d60 2024-03-25T18:29:49.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.594+0000 7fbbb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:49.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.594+0000 7fbbb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:49.596 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:29:49.596 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:49.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 10 librados: watch_flush enter 2024-03-25T18:29:49.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 10 librados: watch_flush exit 2024-03-25T18:29:49.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T18:29:49.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 10 client.4404.objecter successfully canceled tick 2024-03-25T18:29:49.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 >> v1:172.21.15.146:6800/3848252714 conn(0x7fbba406de10 legacy=0x7fbba40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:49.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 10 monclient: shutdown 2024-03-25T18:29:49.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:49.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 >> v1:172.21.15.102:6789/0 conn(0x7fbbc00b4d60 legacy=0x7fbbc0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:49.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 shutdown_connections 2024-03-25T18:29:49.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 >> 172.21.15.102:0/7577185 conn(0x7fbbc00617a0 msgr2=0x7fbbc013c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:49.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 shutdown_connections 2024-03-25T18:29:49.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 1 -- 172.21.15.102:0/7577185 wait complete. 2024-03-25T18:29:49.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:49.596+0000 7fbbc7560640 1 librados: shutdown 2024-03-25T18:29:49.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:56.417 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:56.531 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.529+0000 7f3edb6ba640 10 monclient: get_monmap_and_config 2024-03-25T18:29:56.531 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.529+0000 7f3edb6ba640 10 monclient: build_initial_monmap 2024-03-25T18:29:56.531 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.529+0000 7f3edb6ba640 10 monclient: monmap: 2024-03-25T18:29:56.531 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:56.531 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:56.531 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:29:56.531402+0000 2024-03-25T18:29:56.532 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:29:56.531402+0000 2024-03-25T18:29:56.532 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:56.532 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:56.532 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:56.532 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:29:56.532 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:29:56.532 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:56.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.530+0000 7f3edb6ba640 1 Processor -- start 2024-03-25T18:29:56.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.530+0000 7f3edb6ba640 1 -- start start 2024-03-25T18:29:56.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.530+0000 7f3edb6ba640 10 monclient: init 2024-03-25T18:29:56.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:56.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:56.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 10 monclient(hunting): picked mon.noname-c con 0x7f3ed40b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:56.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 10 monclient(hunting): picked mon.noname-b con 0x7f3ed40b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:56.534 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 10 monclient(hunting): picked mon.noname-a con 0x7f3ed414be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:56.534 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed40af940 con 0x7f3ed414be90 2024-03-25T18:29:56.534 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed40b0a70 con 0x7f3ed40b4d60 2024-03-25T18:29:56.534 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed40b0d50 con 0x7f3ed40b1730 2024-03-25T18:29:56.534 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:56.534 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.531+0000 7f3edb6ba640 10 monclient(hunting): authenticate will time out at 7627.820373s 2024-03-25T18:29:56.540 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.538+0000 7f3ed9c57640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3ed414be90 0x7f3ed414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44024/0 (socket says 172.21.15.102:44024) 2024-03-25T18:29:56.540 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.538+0000 7f3ed9c57640 1 -- 172.21.15.102:0/483368227 learned_addr learned my addr 172.21.15.102:0/483368227 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:56.554 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.552+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368639580 0 0) 0x7f3ed40af940 con 0x7f3ed414be90 2024-03-25T18:29:56.554 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.552+0000 7f3ecbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:56.554 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.552+0000 7f3ecbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:56.554 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.552+0000 7f3ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:56.554 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.552+0000 7f3ecbfff640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:29:56.555 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.552+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec0003520 con 0x7f3ed414be90 2024-03-25T18:29:56.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.556+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633289241 0 0) 0x7f3ed40b0d50 con 0x7f3ed40b1730 2024-03-25T18:29:56.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.556+0000 7f3ecbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:56.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.556+0000 7f3ecbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:56.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.556+0000 7f3ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:56.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.556+0000 7f3ecbfff640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:29:56.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.556+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed40af940 con 0x7f3ed40b1730 2024-03-25T18:29:56.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.559+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011899277 0 0) 0x7f3ed40b0a70 con 0x7f3ed40b4d60 2024-03-25T18:29:56.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.559+0000 7f3ecbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:56.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.559+0000 7f3ecbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:56.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.559+0000 7f3ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:56.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.559+0000 7f3ecbfff640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:29:56.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.559+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed40b0d50 con 0x7f3ed40b4d60 2024-03-25T18:29:56.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3418303123 0 0) 0x7f3ec0003520 con 0x7f3ed414be90 2024-03-25T18:29:56.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 >> v1:172.21.15.102:6790/0 conn(0x7f3ed40b1730 legacy=0x7f3ed40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 >> v1:172.21.15.146:6789/0 conn(0x7f3ed40b4d60 legacy=0x7f3ed413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 1 monclient: found mon.noname-a 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed40b1030 con 0x7f3ed414be90 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 10 monclient: _finish_auth 0 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3ecc001a10 con 0x7f3ed414be90 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.561+0000 7f3ecbfff640 10 monclient: dump: 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:56.564 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.562+0000 7f3edb6ba640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:29:56.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.562+0000 7f3edb6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:56.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.565+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ecc001b70 con 0x7f3ed414be90 2024-03-25T18:29:56.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.565+0000 7f3ecbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:56.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.565+0000 7f3ecbfff640 1 -- 172.21.15.102:0/483368227 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3ecc001f80 con 0x7f3ed414be90 2024-03-25T18:29:56.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.565+0000 7f3edb6ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:56.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.565+0000 7f3edb6ba640 4 set_mon_vals no callback set 2024-03-25T18:29:56.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.565+0000 7f3ecbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:56.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.565+0000 7f3ecbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:56.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.565+0000 7f3ecbfff640 10 monclient: dump: 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:56.568 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:56.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.566+0000 7f3edb6ba640 10 monclient: shutdown 2024-03-25T18:29:56.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.566+0000 7f3edb6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.566+0000 7f3edb6ba640 1 -- 172.21.15.102:0/483368227 >> v1:172.21.15.102:6789/0 conn(0x7f3ed414be90 legacy=0x7f3ed414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.566+0000 7f3edb6ba640 1 -- 172.21.15.102:0/483368227 shutdown_connections 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.566+0000 7f3edb6ba640 1 -- 172.21.15.102:0/483368227 >> 172.21.15.102:0/483368227 conn(0x7f3ed40617a0 msgr2=0x7f3ed4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.566+0000 7f3edb6ba640 1 -- 172.21.15.102:0/483368227 shutdown_connections 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.566+0000 7f3edb6ba640 1 -- 172.21.15.102:0/483368227 wait complete. 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 10 monclient: build_initial_monmap 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 10 monclient: monmap: 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:29:56.571 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 1 librados: starting msgr at 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 1 librados: starting objecter 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 1 Processor -- start 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 1 -- start start 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 1 librados: setting wanted keys 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 1 librados: calling monclient init 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.567+0000 7f3edb6ba640 10 monclient: init 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 10 monclient(hunting): picked mon.noname-b con 0x7f3ed40b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:29:56.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 10 monclient(hunting): picked mon.noname-c con 0x7f3ed40b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:29:56.573 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 10 monclient(hunting): picked mon.noname-a con 0x7f3ed414be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:29:56.573 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed4142710 con 0x7f3ed414be90 2024-03-25T18:29:56.573 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed4142410 con 0x7f3ed40b1730 2024-03-25T18:29:56.573 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed4140d00 con 0x7f3ed40b4d60 2024-03-25T18:29:56.573 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:56.573 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.568+0000 7f3edb6ba640 10 monclient(hunting): authenticate will time out at 7627.857599s 2024-03-25T18:29:56.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.578+0000 7f3ed9c57640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3ed414be90 0x7f3ed41e2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44058/0 (socket says 172.21.15.102:44058) 2024-03-25T18:29:56.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.578+0000 7f3ed9c57640 1 -- 172.21.15.102:0/2984929933 learned_addr learned my addr 172.21.15.102:0/2984929933 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:29:56.581 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.579+0000 7f3ec9ffb640 10 client.?.objecter ms_handle_connect 0x7f3ed40b1730 2024-03-25T18:29:56.581 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.579+0000 7f3ec9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:56.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.582+0000 7f3ec9ffb640 10 client.?.objecter ms_handle_connect 0x7f3ed40b4d60 2024-03-25T18:29:56.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.582+0000 7f3ec9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:56.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.585+0000 7f3ec9ffb640 10 client.?.objecter ms_handle_connect 0x7f3ed414be90 2024-03-25T18:29:56.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.585+0000 7f3ec9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:56.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.594+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065950145 0 0) 0x7f3ed4142710 con 0x7f3ed414be90 2024-03-25T18:29:56.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.594+0000 7f3ec9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:56.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.594+0000 7f3ec9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:56.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.594+0000 7f3ec9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:56.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.594+0000 7f3ec9ffb640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:29:56.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.594+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb8004eb0 con 0x7f3ed414be90 2024-03-25T18:29:56.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.599+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635825349 0 0) 0x7f3ed4142410 con 0x7f3ed40b1730 2024-03-25T18:29:56.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.599+0000 7f3ec9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:56.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.599+0000 7f3ec9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:56.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.599+0000 7f3ec9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:56.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.599+0000 7f3ec9ffb640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:29:56.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.599+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed4142710 con 0x7f3ed40b1730 2024-03-25T18:29:56.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.602+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318120745 0 0) 0x7f3ed4140d00 con 0x7f3ed40b4d60 2024-03-25T18:29:56.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.602+0000 7f3ec9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:56.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.602+0000 7f3ec9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:56.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.602+0000 7f3ec9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:56.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.602+0000 7f3ec9ffb640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:29:56.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.602+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed4142410 con 0x7f3ed40b4d60 2024-03-25T18:29:56.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.607+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 32371670 0 0) 0x7f3eb8004eb0 con 0x7f3ed414be90 2024-03-25T18:29:56.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.607+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed4140d00 con 0x7f3ed414be90 2024-03-25T18:29:56.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.607+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3ecc001ec0 con 0x7f3ed414be90 2024-03-25T18:29:56.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.607+0000 7f3ec9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:56.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.607+0000 7f3ec9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:56.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.607+0000 7f3ec9ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:56.610 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:56.610 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:56.610 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.610 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.611 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:56.611 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:56.611 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:56.611 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:56.611 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:56.611 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:56.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.607+0000 7f3ec9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:56.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.611+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2552057630 0 0) 0x7f3ed4142710 con 0x7f3ed40b1730 2024-03-25T18:29:56.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.611+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3eb8004eb0 con 0x7f3ed40b1730 2024-03-25T18:29:56.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.611+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3ed0001870 con 0x7f3ed40b1730 2024-03-25T18:29:56.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.611+0000 7f3ec9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:56.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.611+0000 7f3ec9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.611+0000 7f3ec9ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:56.614 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:56.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.616+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2314071480 0 0) 0x7f3ed4142410 con 0x7f3ed40b4d60 2024-03-25T18:29:56.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.616+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed4142710 con 0x7f3ed40b4d60 2024-03-25T18:29:56.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.616+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3ec40018f0 con 0x7f3ed40b4d60 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.616+0000 7f3ec9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.616+0000 7f3ec9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.616+0000 7f3ec9ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:56.619 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:56.620 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:56.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 701881563 0 0) 0x7f3ed4140d00 con 0x7f3ed414be90 2024-03-25T18:29:56.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 >> v1:172.21.15.102:6790/0 conn(0x7f3ed40b4d60 legacy=0x7f3ed4142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 >> v1:172.21.15.146:6789/0 conn(0x7f3ed40b1730 legacy=0x7f3ed41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:56.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 1 monclient: found mon.a 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed4139000 con 0x7f3ed414be90 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3ec9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3edb6ba640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3edb6ba640 10 monclient: _renew_subs 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3edb6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.625+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed4141c30 con 0x7f3ed414be90 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.626+0000 7f3edb6ba640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.626+0000 7f3edb6ba640 10 monclient: _renew_subs 2024-03-25T18:29:56.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.626+0000 7f3edb6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:56.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.626+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed413b9d0 con 0x7f3ed414be90 2024-03-25T18:29:56.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.626+0000 7f3edb6ba640 1 librados: init done 2024-03-25T18:29:56.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.627+0000 7f3edb6ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:56.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.627+0000 7f3edb6ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:56.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.627+0000 7f3edb6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:56.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.627+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed41428c0 con 0x7f3ed414be90 2024-03-25T18:29:56.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.936+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ecc0020e0 con 0x7f3ed414be90 2024-03-25T18:29:56.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.936+0000 7f3ec9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:56.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.936+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3ecc004fb0 con 0x7f3ed414be90 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.936+0000 7f3ec9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.936+0000 7f3ec9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.936+0000 7f3ec9ffb640 10 monclient: dump: 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:29:56.939 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.936+0000 7f3ecb7fe640 4 set_mon_vals no callback set 2024-03-25T18:29:56.940 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.937+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f3ecc01af80 con 0x7f3ed414be90 2024-03-25T18:29:56.940 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.938+0000 7f3ed9456640 10 monclient: get_auth_request con 0x7f3eb806e0f0 auth_method 0 2024-03-25T18:29:56.940 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.938+0000 7f3ec9ffb640 10 client.4410.objecter ms_handle_connect 0x7f3eb806e0f0 2024-03-25T18:29:56.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.943+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f3ecc054a20 con 0x7f3ed414be90 2024-03-25T18:29:56.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.943+0000 7f3ec9ffb640 10 client.4410.objecter ms_dispatch 0x7f3ed4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:56.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.943+0000 7f3ec9ffb640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:56.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.943+0000 7f3ec9ffb640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:56.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.943+0000 7f3ec9ffb640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T18:29:56.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.943+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ecc050360 con 0x7f3ed414be90 2024-03-25T18:29:56.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.943+0000 7f3ec9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:56.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:56.943+0000 7f3ec9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:57.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.189+0000 7f3edb6ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:57.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.189+0000 7f3edb6ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:57.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.189+0000 7f3edb6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:29:57.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.189+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ed40627b0 con 0x7f3ed414be90 2024-03-25T18:29:57.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.216+0000 7f3ec9ffb640 1 -- 172.21.15.102:0/2984929933 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f3ecc052460 con 0x7f3ed414be90 2024-03-25T18:29:57.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.216+0000 7f3ec9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:57.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.216+0000 7f3ec9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:57.219 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:29:57.219 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.218+0000 7f3edb6ba640 10 librados: watch_flush enter 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.218+0000 7f3edb6ba640 10 librados: watch_flush exit 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.218+0000 7f3edb6ba640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.218+0000 7f3edb6ba640 10 client.4410.objecter successfully canceled tick 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 >> v1:172.21.15.146:6800/3848252714 conn(0x7f3eb806e0f0 legacy=0x7f3eb80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 10 monclient: shutdown 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 >> v1:172.21.15.102:6789/0 conn(0x7f3ed414be90 legacy=0x7f3ed41e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 shutdown_connections 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 >> 172.21.15.102:0/2984929933 conn(0x7f3ed40617a0 msgr2=0x7f3ed40b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 shutdown_connections 2024-03-25T18:29:57.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 1 -- 172.21.15.102:0/2984929933 wait complete. 2024-03-25T18:29:57.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:29:57.219+0000 7f3edb6ba640 1 librados: shutdown 2024-03-25T18:29:57.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:04.039 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:04.154 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.152+0000 7f0ccb20e640 10 monclient: get_monmap_and_config 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.152+0000 7f0ccb20e640 10 monclient: build_initial_monmap 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.152+0000 7f0ccb20e640 10 monclient: monmap: 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:30:04.154489+0000 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:30:04.154489+0000 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:30:04.155 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:04.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.153+0000 7f0ccb20e640 1 Processor -- start 2024-03-25T18:30:04.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.153+0000 7f0ccb20e640 1 -- start start 2024-03-25T18:30:04.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.153+0000 7f0ccb20e640 10 monclient: init 2024-03-25T18:30:04.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:04.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:04.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 10 monclient(hunting): picked mon.noname-b con 0x7f0cc40a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:04.157 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 10 monclient(hunting): picked mon.noname-c con 0x7f0cc413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:04.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc41430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:04.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc4142520 con 0x7f0cc41430c0 2024-03-25T18:30:04.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc40b28a0 con 0x7f0cc413f710 2024-03-25T18:30:04.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc40b2a30 con 0x7f0cc40a98f0 2024-03-25T18:30:04.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:04.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.154+0000 7f0ccb20e640 10 monclient(hunting): authenticate will time out at 7635.443570s 2024-03-25T18:30:04.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.158+0000 7f0cc8faa640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0cc40a98f0 0x7f0cc40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:50186/0 (socket says 172.21.15.102:50186) 2024-03-25T18:30:04.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.158+0000 7f0cc8faa640 1 -- 172.21.15.102:0/355318763 learned_addr learned my addr 172.21.15.102:0/355318763 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:04.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.179+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808961304 0 0) 0x7f0cc40b2a30 con 0x7f0cc40a98f0 2024-03-25T18:30:04.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.179+0000 7f0cc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:04.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.179+0000 7f0cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:04.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.179+0000 7f0cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:04.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.179+0000 7f0cc37fe640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:30:04.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.179+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb4003520 con 0x7f0cc40a98f0 2024-03-25T18:30:04.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.182+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70985344 0 0) 0x7f0cc4142520 con 0x7f0cc41430c0 2024-03-25T18:30:04.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.182+0000 7f0cc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:04.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.182+0000 7f0cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:04.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.182+0000 7f0cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:04.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.182+0000 7f0cc37fe640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:30:04.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.182+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc40b2a30 con 0x7f0cc41430c0 2024-03-25T18:30:04.186 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.184+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991776648 0 0) 0x7f0cc40b28a0 con 0x7f0cc413f710 2024-03-25T18:30:04.186 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:04.186 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:04.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:04.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:30:04.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc4142520 con 0x7f0cc413f710 2024-03-25T18:30:04.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1049533249 0 0) 0x7f0cb4003520 con 0x7f0cc40a98f0 2024-03-25T18:30:04.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 >> v1:172.21.15.146:6789/0 conn(0x7f0cc413f710 legacy=0x7f0cc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 >> v1:172.21.15.102:6789/0 conn(0x7f0cc41430c0 legacy=0x7f0cc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:04.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 1 monclient: found mon.noname-b 2024-03-25T18:30:04.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:04.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc40b44c0 con 0x7f0cc40a98f0 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0cb8001690 con 0x7f0cc40a98f0 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0cc37fe640 10 monclient: dump: 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:04.189 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:04.190 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:04.190 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:04.190 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:04.190 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:04.190 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:04.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0ccb20e640 5 monclient: authenticate success, global_id 4397 2024-03-25T18:30:04.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.185+0000 7f0ccb20e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.189+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cb80017f0 con 0x7f0cc40a98f0 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.189+0000 7f0cc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.189+0000 7f0cc37fe640 1 -- 172.21.15.102:0/355318763 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0cb8001c00 con 0x7f0cc40a98f0 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.189+0000 7f0cc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.189+0000 7f0cc37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.189+0000 7f0cc37fe640 10 monclient: dump: 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:04.191 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:04.192 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:04.192 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:04.192 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:04.192 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:04.192 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:04.192 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.189+0000 7f0ccb20e640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:04.192 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.189+0000 7f0ccb20e640 4 set_mon_vals no callback set 2024-03-25T18:30:04.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.190+0000 7f0ccb20e640 10 monclient: shutdown 2024-03-25T18:30:04.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.190+0000 7f0ccb20e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:04.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.190+0000 7f0ccb20e640 1 -- 172.21.15.102:0/355318763 >> v1:172.21.15.102:6790/0 conn(0x7f0cc40a98f0 legacy=0x7f0cc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.190+0000 7f0ccb20e640 1 -- 172.21.15.102:0/355318763 shutdown_connections 2024-03-25T18:30:04.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.190+0000 7f0ccb20e640 1 -- 172.21.15.102:0/355318763 >> 172.21.15.102:0/355318763 conn(0x7f0cc40b7ea0 msgr2=0x7f0cc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:04.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.190+0000 7f0ccb20e640 1 -- 172.21.15.102:0/355318763 shutdown_connections 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.190+0000 7f0ccb20e640 1 -- 172.21.15.102:0/355318763 wait complete. 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.191+0000 7f0ccb20e640 10 monclient: build_initial_monmap 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.191+0000 7f0ccb20e640 10 monclient: monmap: 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:30:04.195 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.191+0000 7f0ccb20e640 1 librados: starting msgr at 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.191+0000 7f0ccb20e640 1 librados: starting objecter 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.191+0000 7f0ccb20e640 1 Processor -- start 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 1 -- start start 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 1 librados: setting wanted keys 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 1 librados: calling monclient init 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 10 monclient: init 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 10 monclient(hunting): picked mon.noname-b con 0x7f0cc40a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 10 monclient(hunting): picked mon.noname-c con 0x7f0cc413f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc41430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:04.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc4137720 con 0x7f0cc41430c0 2024-03-25T18:30:04.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc41dc070 con 0x7f0cc40a98f0 2024-03-25T18:30:04.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc4139140 con 0x7f0cc413f710 2024-03-25T18:30:04.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:04.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.192+0000 7f0ccb20e640 10 monclient(hunting): authenticate will time out at 7635.481789s 2024-03-25T18:30:04.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.201+0000 7f0cc3fff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f0cc413f710 0x7f0cc41db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:50232/0 (socket says 172.21.15.102:50232) 2024-03-25T18:30:04.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.201+0000 7f0cc3fff640 1 -- 172.21.15.102:0/4004893195 learned_addr learned my addr 172.21.15.102:0/4004893195 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:04.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.203+0000 7f0cc17fa640 10 client.?.objecter ms_handle_connect 0x7f0cc413f710 2024-03-25T18:30:04.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.203+0000 7f0cc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:04.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.205+0000 7f0cc17fa640 10 client.?.objecter ms_handle_connect 0x7f0cc41430c0 2024-03-25T18:30:04.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.205+0000 7f0cc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:04.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.214+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724299141 0 0) 0x7f0cc4139140 con 0x7f0cc413f710 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca8004eb0 con 0x7f0cc413f710 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946485775 0 0) 0x7f0cc4137720 con 0x7f0cc41430c0 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:30:04.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.215+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc4139140 con 0x7f0cc41430c0 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.223+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4053757604 0 0) 0x7f0ca8004eb0 con 0x7f0cc413f710 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.223+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cc4137720 con 0x7f0cc413f710 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.223+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0cb0001a10 con 0x7f0cc413f710 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.223+0000 7f0cc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.223+0000 7f0cc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.223+0000 7f0cc17fa640 10 monclient(hunting): dump: 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:04.225 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.223+0000 7f0cc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.224+0000 7f0cc17fa640 10 client.?.objecter ms_handle_connect 0x7f0cc40a98f0 2024-03-25T18:30:04.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.224+0000 7f0cc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:04.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3900609966 0 0) 0x7f0cc4137720 con 0x7f0cc413f710 2024-03-25T18:30:04.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 >> v1:172.21.15.146:6789/0 conn(0x7f0cc40a98f0 legacy=0x7f0cc4137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 >> v1:172.21.15.102:6789/0 conn(0x7f0cc41430c0 legacy=0x7f0cc41db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 1 monclient: found mon.c 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc4139420 con 0x7f0cc413f710 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 10 monclient: _finish_auth 0 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0cc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 5 monclient: authenticate success, global_id 4400 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 10 monclient: _renew_subs 2024-03-25T18:30:04.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:04.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc4136dc0 con 0x7f0cc413f710 2024-03-25T18:30:04.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:04.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 10 monclient: _renew_subs 2024-03-25T18:30:04.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:04.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc4139840 con 0x7f0cc413f710 2024-03-25T18:30:04.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.236+0000 7f0ccb20e640 1 librados: init done 2024-03-25T18:30:04.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.238+0000 7f0ccb20e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:04.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.238+0000 7f0ccb20e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:04.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.238+0000 7f0ccb20e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:04.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.238+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc414f2d0 con 0x7f0cc413f710 2024-03-25T18:30:04.550 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.548+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cb0001c70 con 0x7f0cc413f710 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.548+0000 7f0cc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.548+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0cb0003a60 con 0x7f0cc413f710 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.548+0000 7f0cc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.548+0000 7f0cc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.548+0000 7f0cc17fa640 10 monclient: dump: 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:04.551 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:04.552 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:04.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.548+0000 7f0cc2ffd640 4 set_mon_vals no callback set 2024-03-25T18:30:04.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.549+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f0cb0003c80 con 0x7f0cc413f710 2024-03-25T18:30:04.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.550+0000 7f0cc8faa640 10 monclient: get_auth_request con 0x7f0ca806dd30 auth_method 0 2024-03-25T18:30:04.552 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.550+0000 7f0cc17fa640 10 client.4400.objecter ms_handle_connect 0x7f0ca806dd30 2024-03-25T18:30:04.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.559+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f0cb0054860 con 0x7f0cc413f710 2024-03-25T18:30:04.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.559+0000 7f0cc17fa640 10 client.4400.objecter ms_dispatch 0x7f0cc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:04.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.559+0000 7f0cc17fa640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:04.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.559+0000 7f0cc17fa640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:04.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.559+0000 7f0cc17fa640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-25T18:30:04.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.559+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cb0050370 con 0x7f0cc413f710 2024-03-25T18:30:04.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.559+0000 7f0cc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:04.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.559+0000 7f0cc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:04.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.776+0000 7f0ccb20e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:04.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.776+0000 7f0ccb20e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:04.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.776+0000 7f0ccb20e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:04.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.776+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cc4136f50 con 0x7f0cc413f710 2024-03-25T18:30:04.860 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.858+0000 7f0cc17fa640 1 -- 172.21.15.102:0/4004893195 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f0cb00522a0 con 0x7f0cc413f710 2024-03-25T18:30:04.860 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.858+0000 7f0cc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:04.860 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.858+0000 7f0cc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:04.861 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:30:04.861 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.860+0000 7f0ccb20e640 10 librados: watch_flush enter 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.860+0000 7f0ccb20e640 10 librados: watch_flush exit 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.860+0000 7f0ccb20e640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.860+0000 7f0ccb20e640 10 client.4400.objecter successfully canceled tick 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.860+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 >> v1:172.21.15.146:6800/3848252714 conn(0x7f0ca806dd30 legacy=0x7f0ca80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.860+0000 7f0ccb20e640 10 monclient: shutdown 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.860+0000 7f0ccb20e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.860+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 >> v1:172.21.15.102:6790/0 conn(0x7f0cc413f710 legacy=0x7f0cc41db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.861+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 shutdown_connections 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.861+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 >> 172.21.15.102:0/4004893195 conn(0x7f0cc40b7ea0 msgr2=0x7f0cc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:04.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.861+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 shutdown_connections 2024-03-25T18:30:04.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.861+0000 7f0ccb20e640 1 -- 172.21.15.102:0/4004893195 wait complete. 2024-03-25T18:30:04.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:04.861+0000 7f0ccb20e640 1 librados: shutdown 2024-03-25T18:30:04.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:11.683 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:11.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.799+0000 7fe46e582640 10 monclient: get_monmap_and_config 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.799+0000 7fe46e582640 10 monclient: build_initial_monmap 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.800+0000 7fe46e582640 10 monclient: monmap: 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:30:11.800652+0000 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:30:11.800652+0000 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:30:11.801 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:11.802 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.800+0000 7fe46e582640 1 Processor -- start 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.800+0000 7fe46e582640 1 -- start start 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.800+0000 7fe46e582640 10 monclient: init 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 10 monclient(hunting): picked mon.noname-b con 0x7fe4680b0070 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 10 monclient(hunting): picked mon.noname-a con 0x7fe46813c170 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 10 monclient(hunting): picked mon.noname-c con 0x7fe46806fa90 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4680b0450 con 0x7fe46813c170 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe468071440 con 0x7fe46806fa90 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4680b0bf0 con 0x7fe4680b0070 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:11.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.801+0000 7fe46e582640 10 monclient(hunting): authenticate will time out at 7643.089884s 2024-03-25T18:30:11.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.810+0000 7fe4677fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe46813c170 0x7fe468141d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54634/0 (socket says 172.21.15.102:54634) 2024-03-25T18:30:11.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.810+0000 7fe4677fe640 1 -- 172.21.15.102:0/3266349448 learned_addr learned my addr 172.21.15.102:0/3266349448 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:11.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.817+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876013607 0 0) 0x7fe4680b0450 con 0x7fe46813c170 2024-03-25T18:30:11.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.817+0000 7fe466ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:11.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.817+0000 7fe466ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:11.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.817+0000 7fe466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:11.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.817+0000 7fe466ffd640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:30:11.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.817+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe450003520 con 0x7fe46813c170 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931902068 0 0) 0x7fe468071440 con 0x7fe46806fa90 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:30:11.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4680b0450 con 0x7fe46806fa90 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977161605 0 0) 0x7fe450003520 con 0x7fe46813c170 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 >> v1:172.21.15.102:6790/0 conn(0x7fe4680b0070 legacy=0x7fe46813ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 >> v1:172.21.15.146:6789/0 conn(0x7fe46806fa90 legacy=0x7fe46814a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 1 monclient: found mon.noname-a 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4680716b0 con 0x7fe46813c170 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:11.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:11.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe466ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:11.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe46e582640 5 monclient: authenticate success, global_id 4419 2024-03-25T18:30:11.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.824+0000 7fe46e582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:11.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.839+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe45c001650 con 0x7fe46813c170 2024-03-25T18:30:11.840 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.839+0000 7fe466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe466ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe466ffd640 10 monclient: dump: 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:11.841 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe45c0018a0 con 0x7fe46813c170 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe46e582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe466ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe46e582640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe466ffd640 1 -- 172.21.15.102:0/3266349448 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe45c001c30 con 0x7fe46813c170 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe46e582640 4 set_mon_vals no callback set 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe466ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe466ffd640 10 monclient: dump: 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:11.842 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:11.843 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:11.843 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:11.843 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:11.843 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:11.843 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe46e582640 10 monclient: shutdown 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe46e582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.840+0000 7fe46e582640 1 -- 172.21.15.102:0/3266349448 >> v1:172.21.15.102:6789/0 conn(0x7fe46813c170 legacy=0x7fe468141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.841+0000 7fe46e582640 1 -- 172.21.15.102:0/3266349448 shutdown_connections 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.841+0000 7fe46e582640 1 -- 172.21.15.102:0/3266349448 >> 172.21.15.102:0/3266349448 conn(0x7fe4680b0ef0 msgr2=0x7fe4680b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.841+0000 7fe46e582640 1 -- 172.21.15.102:0/3266349448 shutdown_connections 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.841+0000 7fe46e582640 1 -- 172.21.15.102:0/3266349448 wait complete. 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 10 monclient: build_initial_monmap 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 10 monclient: monmap: 2024-03-25T18:30:11.845 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 1 librados: starting msgr at 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 1 librados: starting objecter 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 1 Processor -- start 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 1 -- start start 2024-03-25T18:30:11.846 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 1 librados: setting wanted keys 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 1 librados: calling monclient init 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.842+0000 7fe46e582640 10 monclient: init 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.843+0000 7fe46e582640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.843+0000 7fe46e582640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.843+0000 7fe46e582640 10 monclient(hunting): picked mon.noname-c con 0x7fe46806fa90 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.843+0000 7fe46e582640 10 monclient(hunting): picked mon.noname-b con 0x7fe4680b0070 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.843+0000 7fe46e582640 10 monclient(hunting): picked mon.noname-a con 0x7fe46813c170 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.844+0000 7fe46e582640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe468128540 con 0x7fe46813c170 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.844+0000 7fe46e582640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe46813d220 con 0x7fe4680b0070 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.844+0000 7fe46e582640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe46813c630 con 0x7fe46806fa90 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.844+0000 7fe46e582640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:11.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.844+0000 7fe46e582640 10 monclient(hunting): authenticate will time out at 7643.132262s 2024-03-25T18:30:11.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.855+0000 7fe467fff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe46806fa90 0x7fe468140cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:38178/0 (socket says 172.21.15.102:38178) 2024-03-25T18:30:11.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.855+0000 7fe467fff640 1 -- 172.21.15.102:0/2078600484 learned_addr learned my addr 172.21.15.102:0/2078600484 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:11.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.857+0000 7fe464ff9640 10 client.?.objecter ms_handle_connect 0x7fe4680b0070 2024-03-25T18:30:11.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.857+0000 7fe464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:11.859 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.858+0000 7fe464ff9640 10 client.?.objecter ms_handle_connect 0x7fe46813c170 2024-03-25T18:30:11.859 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.858+0000 7fe464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:11.860 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.860+0000 7fe464ff9640 10 client.?.objecter ms_handle_connect 0x7fe46806fa90 2024-03-25T18:30:11.860 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.860+0000 7fe464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:11.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.867+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484447442 0 0) 0x7fe4680b0450 con 0x7fe46813c170 2024-03-25T18:30:11.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.867+0000 7fe464ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:11.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.867+0000 7fe464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:11.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.867+0000 7fe464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:11.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.867+0000 7fe464ff9640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:30:11.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.867+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe454004eb0 con 0x7fe46813c170 2024-03-25T18:30:11.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.869+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881595991 0 0) 0x7fe46813c630 con 0x7fe46806fa90 2024-03-25T18:30:11.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.869+0000 7fe464ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:11.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.869+0000 7fe464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:11.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.869+0000 7fe464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:11.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.869+0000 7fe464ff9640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:30:11.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.869+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4680b0450 con 0x7fe46806fa90 2024-03-25T18:30:11.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.871+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936013026 0 0) 0x7fe46813d220 con 0x7fe4680b0070 2024-03-25T18:30:11.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.871+0000 7fe464ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:11.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.871+0000 7fe464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:11.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.871+0000 7fe464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:11.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.871+0000 7fe464ff9640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:30:11.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.871+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe46813c630 con 0x7fe4680b0070 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.882+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228510177 0 0) 0x7fe4680b0450 con 0x7fe46806fa90 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.882+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe46813d220 con 0x7fe46806fa90 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.882+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe458001970 con 0x7fe46806fa90 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.882+0000 7fe464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.882+0000 7fe464ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.882+0000 7fe464ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:11.883 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:11.884 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:11.884 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:11.884 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:11.884 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:11.884 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:11.884 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:11.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.882+0000 7fe464ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:11.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2086512315 0 0) 0x7fe46813d220 con 0x7fe46806fa90 2024-03-25T18:30:11.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 >> v1:172.21.15.146:6789/0 conn(0x7fe4680b0070 legacy=0x7fe4680b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 >> v1:172.21.15.102:6789/0 conn(0x7fe46813c170 legacy=0x7fe4681e0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 1 monclient: found mon.c 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468128fd0 con 0x7fe46806fa90 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe464ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe46e582640 5 monclient: authenticate success, global_id 4403 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe46e582640 10 monclient: _renew_subs 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe46e582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.894+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe468140170 con 0x7fe46806fa90 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.895+0000 7fe46e582640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:11.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.895+0000 7fe46e582640 10 monclient: _renew_subs 2024-03-25T18:30:11.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.895+0000 7fe46e582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:11.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.895+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe468126810 con 0x7fe46806fa90 2024-03-25T18:30:11.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.895+0000 7fe46e582640 1 librados: init done 2024-03-25T18:30:11.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.895+0000 7fe46e582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:11.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.896+0000 7fe46e582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:11.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.896+0000 7fe46e582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:11.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:11.896+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4681291c0 con 0x7fe46806fa90 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.187+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4580035f0 con 0x7fe46806fa90 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.187+0000 7fe464ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.187+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe458003980 con 0x7fe46806fa90 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.187+0000 7fe464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.187+0000 7fe464ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.187+0000 7fe464ff9640 10 monclient: dump: 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:12.188 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:12.189 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:12.189 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:12.189 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:12.189 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:12.189 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:12.189 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:12.189 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:12.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.187+0000 7fe4667fc640 4 set_mon_vals no callback set 2024-03-25T18:30:12.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.188+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fe458003ba0 con 0x7fe46806fa90 2024-03-25T18:30:12.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.189+0000 7fe4677fe640 10 monclient: get_auth_request con 0x7fe45406de20 auth_method 0 2024-03-25T18:30:12.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.189+0000 7fe464ff9640 10 client.4403.objecter ms_handle_connect 0x7fe45406de20 2024-03-25T18:30:12.195 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.194+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fe458054750 con 0x7fe46806fa90 2024-03-25T18:30:12.195 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.194+0000 7fe464ff9640 10 client.4403.objecter ms_dispatch 0x7fe468000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:12.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.194+0000 7fe464ff9640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:12.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.194+0000 7fe464ff9640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:12.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.195+0000 7fe464ff9640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-25T18:30:12.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.195+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe458050260 con 0x7fe46806fa90 2024-03-25T18:30:12.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.195+0000 7fe464ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:12.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.195+0000 7fe464ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:12.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.422+0000 7fe46e582640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:12.424 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.423+0000 7fe46e582640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:12.424 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.423+0000 7fe46e582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:12.424 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.423+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4680627b0 con 0x7fe46806fa90 2024-03-25T18:30:12.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.459+0000 7fe464ff9640 1 -- 172.21.15.102:0/2078600484 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fe458052190 con 0x7fe46806fa90 2024-03-25T18:30:12.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.459+0000 7fe464ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:12.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.459+0000 7fe464ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:12.460 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:30:12.460 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.461+0000 7fe46e582640 10 librados: watch_flush enter 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.461+0000 7fe46e582640 10 librados: watch_flush exit 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.461+0000 7fe46e582640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.461+0000 7fe46e582640 10 client.4403.objecter successfully canceled tick 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 >> v1:172.21.15.146:6800/3848252714 conn(0x7fe45406de20 legacy=0x7fe4540702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 10 monclient: shutdown 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 >> v1:172.21.15.102:6790/0 conn(0x7fe46806fa90 legacy=0x7fe468140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 shutdown_connections 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 >> 172.21.15.102:0/2078600484 conn(0x7fe4680b0ef0 msgr2=0x7fe46814ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 shutdown_connections 2024-03-25T18:30:12.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 1 -- 172.21.15.102:0/2078600484 wait complete. 2024-03-25T18:30:12.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:12.462+0000 7fe46e582640 1 librados: shutdown 2024-03-25T18:30:12.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:19.282 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.396+0000 7fdb40d0d640 10 monclient: get_monmap_and_config 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.396+0000 7fdb40d0d640 10 monclient: build_initial_monmap 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.396+0000 7fdb40d0d640 10 monclient: monmap: 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:30:19.396826+0000 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:30:19.396826+0000 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:30:19.397 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:30:19.398 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 1 Processor -- start 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 1 -- start start 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 10 monclient: init 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb3c0a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb3c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3c1430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.397+0000 7fdb40d0d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c142520 con 0x7fdb3c13f710 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.398+0000 7fdb40d0d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0b28a0 con 0x7fdb3c1430c0 2024-03-25T18:30:19.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.398+0000 7fdb40d0d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0b2a30 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.398+0000 7fdb40d0d640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:19.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.398+0000 7fdb40d0d640 10 monclient(hunting): authenticate will time out at 7650.685961s 2024-03-25T18:30:19.413 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.412+0000 7fdb3a59c640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fdb3c0a98f0 0x7fdb3c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:42940/0 (socket says 172.21.15.102:42940) 2024-03-25T18:30:19.413 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.412+0000 7fdb3a59c640 1 -- 172.21.15.102:0/3239656422 learned_addr learned my addr 172.21.15.102:0/3239656422 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:19.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.429+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703651472 0 0) 0x7fdb3c0b2a30 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.429+0000 7fdb39d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:30:19.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.429+0000 7fdb39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:19.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.429+0000 7fdb39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:19.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.429+0000 7fdb39d9b640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:30:19.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.429+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb28003520 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.433+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961858767 0 0) 0x7fdb3c0b28a0 con 0x7fdb3c1430c0 2024-03-25T18:30:19.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.433+0000 7fdb39d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:30:19.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.433+0000 7fdb39d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:19.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.433+0000 7fdb39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:19.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.433+0000 7fdb39d9b640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:30:19.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.433+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c0b2a30 con 0x7fdb3c1430c0 2024-03-25T18:30:19.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3305642351 0 0) 0x7fdb28003520 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 >> v1:172.21.15.146:6789/0 conn(0x7fdb3c1430c0 legacy=0x7fdb3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 >> v1:172.21.15.102:6789/0 conn(0x7fdb3c13f710 legacy=0x7fdb3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:30:19.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 1 monclient: found mon.noname-b 2024-03-25T18:30:19.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0b44c0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 10 monclient: _finish_auth 0 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdb2c001690 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb39d9b640 10 monclient: dump: 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:19.443 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:19.444 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:19.444 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:19.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb40d0d640 5 monclient: authenticate success, global_id 4406 2024-03-25T18:30:19.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.441+0000 7fdb40d0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:19.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.444+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb2c0017f0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.444+0000 7fdb39d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:19.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.444+0000 7fdb39d9b640 1 -- 172.21.15.102:0/3239656422 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdb2c001c00 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.444+0000 7fdb39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:19.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.444+0000 7fdb39d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:19.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.444+0000 7fdb39d9b640 10 monclient: dump: 2024-03-25T18:30:19.445 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.444+0000 7fdb40d0d640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:19.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.444+0000 7fdb40d0d640 4 set_mon_vals no callback set 2024-03-25T18:30:19.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.445+0000 7fdb40d0d640 10 monclient: shutdown 2024-03-25T18:30:19.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.445+0000 7fdb40d0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:19.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.445+0000 7fdb40d0d640 1 -- 172.21.15.102:0/3239656422 >> v1:172.21.15.102:6790/0 conn(0x7fdb3c0a98f0 legacy=0x7fdb3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.445+0000 7fdb40d0d640 1 -- 172.21.15.102:0/3239656422 shutdown_connections 2024-03-25T18:30:19.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.445+0000 7fdb40d0d640 1 -- 172.21.15.102:0/3239656422 >> 172.21.15.102:0/3239656422 conn(0x7fdb3c0b7ea0 msgr2=0x7fdb3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:19.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.445+0000 7fdb40d0d640 1 -- 172.21.15.102:0/3239656422 shutdown_connections 2024-03-25T18:30:19.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.446+0000 7fdb40d0d640 1 -- 172.21.15.102:0/3239656422 wait complete. 2024-03-25T18:30:19.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 10 monclient: build_initial_monmap 2024-03-25T18:30:19.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 10 monclient: monmap: 2024-03-25T18:30:19.452 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:19.452 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:19.452 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:30:19.452 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:30:19.452 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:19.452 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:30:19.453 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:19.453 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:30:19.453 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:30:19.453 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:19.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 1 librados: starting msgr at 2024-03-25T18:30:19.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 1 librados: starting objecter 2024-03-25T18:30:19.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 1 Processor -- start 2024-03-25T18:30:19.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 1 -- start start 2024-03-25T18:30:19.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 1 librados: setting wanted keys 2024-03-25T18:30:19.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 1 librados: calling monclient init 2024-03-25T18:30:19.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.447+0000 7fdb40d0d640 10 monclient: init 2024-03-25T18:30:19.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:19.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:19.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb3c0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:19.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb3c13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:19.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:19.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c1db7d0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0b1420 con 0x7fdb3c13f710 2024-03-25T18:30:19.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0b15b0 con 0x7fdb3c1430c0 2024-03-25T18:30:19.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:19.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.448+0000 7fdb40d0d640 10 monclient(hunting): authenticate will time out at 7650.736206s 2024-03-25T18:30:19.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.459+0000 7fdb3a59c640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdb3c0a98f0 0x7fdb3c1daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59486/0 (socket says 172.21.15.102:59486) 2024-03-25T18:30:19.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.459+0000 7fdb3a59c640 1 -- 172.21.15.102:0/600656685 learned_addr learned my addr 172.21.15.102:0/600656685 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:19.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.461+0000 7fdb32ffd640 10 client.?.objecter ms_handle_connect 0x7fdb3c0a98f0 2024-03-25T18:30:19.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.461+0000 7fdb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:19.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.467+0000 7fdb32ffd640 10 client.?.objecter ms_handle_connect 0x7fdb3c13f710 2024-03-25T18:30:19.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.467+0000 7fdb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:19.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.474+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229483852 0 0) 0x7fdb3c1db7d0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.474+0000 7fdb32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:19.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.474+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:19.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.474+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:19.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.474+0000 7fdb32ffd640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:30:19.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.474+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb1c004eb0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.474+0000 7fdb32ffd640 10 client.?.objecter ms_handle_connect 0x7fdb3c1430c0 2024-03-25T18:30:19.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.474+0000 7fdb32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:19.480 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.480+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232002256 0 0) 0x7fdb3c0b1420 con 0x7fdb3c13f710 2024-03-25T18:30:19.480 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.480+0000 7fdb32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:19.480 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.480+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:19.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.480+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:19.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.480+0000 7fdb32ffd640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:30:19.481 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.480+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c1db7d0 con 0x7fdb3c13f710 2024-03-25T18:30:19.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.483+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 778695390 0 0) 0x7fdb1c004eb0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.483+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb3c0b1420 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.483+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdb2c001c40 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.483+0000 7fdb32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:19.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.483+0000 7fdb32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:19.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.483+0000 7fdb32ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:19.485 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.484+0000 7fdb32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:19.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.487+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568242370 0 0) 0x7fdb3c0b15b0 con 0x7fdb3c1430c0 2024-03-25T18:30:19.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.487+0000 7fdb32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:19.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.487+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:19.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.487+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:19.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.487+0000 7fdb32ffd640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:30:19.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.487+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb1c004eb0 con 0x7fdb3c1430c0 2024-03-25T18:30:19.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2162876511 0 0) 0x7fdb3c0b1420 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 >> v1:172.21.15.102:6790/0 conn(0x7fdb3c1430c0 legacy=0x7fdb3c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 >> v1:172.21.15.146:6789/0 conn(0x7fdb3c13f710 legacy=0x7fdb3c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:19.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:19.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 1 monclient: found mon.a 2024-03-25T18:30:19.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:19.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0b1740 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 10 monclient: _renew_subs 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c1db1a0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 10 monclient: _renew_subs 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:19.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c0b08f0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.502 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.499+0000 7fdb40d0d640 1 librados: init done 2024-03-25T18:30:19.502 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.500+0000 7fdb40d0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:19.502 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.500+0000 7fdb40d0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:19.502 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.500+0000 7fdb40d0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:19.502 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.500+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c0627b0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.817+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb2c004a90 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.817+0000 7fdb32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:19.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.817+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fdb2c004de0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.817+0000 7fdb32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:19.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.817+0000 7fdb32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:19.818 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.817+0000 7fdb32ffd640 10 monclient: dump: 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:19.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.817+0000 7fdb3959a640 4 set_mon_vals no callback set 2024-03-25T18:30:19.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.818+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fdb2c005000 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.818+0000 7fdb33fff640 10 monclient: get_auth_request con 0x7fdb1c06de00 auth_method 0 2024-03-25T18:30:19.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.819+0000 7fdb32ffd640 10 client.4428.objecter ms_handle_connect 0x7fdb1c06de00 2024-03-25T18:30:19.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.824+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fdb2c054990 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.824+0000 7fdb32ffd640 10 client.4428.objecter ms_dispatch 0x7fdb3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:19.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.824+0000 7fdb32ffd640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:19.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.824+0000 7fdb32ffd640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:19.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.824+0000 7fdb32ffd640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-25T18:30:19.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.824+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb2c0504a0 con 0x7fdb3c0a98f0 2024-03-25T18:30:19.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.824+0000 7fdb32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:19.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:19.824+0000 7fdb32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:20.067 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.066+0000 7fdb40d0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:20.067 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.066+0000 7fdb40d0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:20.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.066+0000 7fdb40d0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:20.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.066+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb3c0abb10 con 0x7fdb3c0a98f0 2024-03-25T18:30:20.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.100+0000 7fdb32ffd640 1 -- 172.21.15.102:0/600656685 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fdb2c0523d0 con 0x7fdb3c0a98f0 2024-03-25T18:30:20.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.100+0000 7fdb32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:20.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.100+0000 7fdb32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:20.101 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:30:20.101 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:20.103 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 10 librados: watch_flush enter 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 10 librados: watch_flush exit 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 10 client.4428.objecter successfully canceled tick 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 >> v1:172.21.15.146:6800/3848252714 conn(0x7fdb1c06de00 legacy=0x7fdb1c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 10 monclient: shutdown 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 >> v1:172.21.15.102:6789/0 conn(0x7fdb3c0a98f0 legacy=0x7fdb3c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 shutdown_connections 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 >> 172.21.15.102:0/600656685 conn(0x7fdb3c0b7ea0 msgr2=0x7fdb3c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 shutdown_connections 2024-03-25T18:30:20.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 1 -- 172.21.15.102:0/600656685 wait complete. 2024-03-25T18:30:20.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:20.102+0000 7fdb40d0d640 1 librados: shutdown 2024-03-25T18:30:20.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:23.617 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:04:39.853 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:23.622 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:04:39.857 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:23.662 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:04:39.898 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:23.668 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:04:39.881 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:23.679 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:04:39.892 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:23.696 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:04:39.932 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:23.708 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:04:39.944 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:23.710 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:04:39.923 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:23.743 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:04:39.979 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:26.922 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:27.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.039+0000 7fd7ffd7b640 10 monclient: get_monmap_and_config 2024-03-25T18:30:27.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.039+0000 7fd7ffd7b640 10 monclient: build_initial_monmap 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.039+0000 7fd7ffd7b640 10 monclient: monmap: 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:30:27.039707+0000 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:30:27.039707+0000 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:30:27.040 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:27.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.039+0000 7fd7ffd7b640 1 Processor -- start 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.039+0000 7fd7ffd7b640 1 -- start start 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.039+0000 7fd7ffd7b640 10 monclient: init 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f80b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f80b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f814be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f80af940 con 0x7fd7f80b1730 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f80b0a70 con 0x7fd7f814be90 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f80b0d50 con 0x7fd7f80b4d60 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:27.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.040+0000 7fd7ffd7b640 10 monclient(hunting): authenticate will time out at 7658.328831s 2024-03-25T18:30:27.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.053+0000 7fd7fdb17640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fd7f80b4d60 0x7fd7f813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:46832/0 (socket says 172.21.15.102:46832) 2024-03-25T18:30:27.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.053+0000 7fd7fdb17640 1 -- 172.21.15.102:0/1217758072 learned_addr learned my addr 172.21.15.102:0/1217758072 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:27.075 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.074+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009648313 0 0) 0x7fd7f80af940 con 0x7fd7f80b1730 2024-03-25T18:30:27.075 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.074+0000 7fd7fcb15640 10 monclient(hunting): _negotiate 2024-03-25T18:30:27.075 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.074+0000 7fd7fcb15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:27.075 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.074+0000 7fd7fcb15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:27.075 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.074+0000 7fd7fcb15640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:30:27.075 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.074+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7e4003520 con 0x7fd7f80b1730 2024-03-25T18:30:27.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.075+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926574097 0 0) 0x7fd7f80b0a70 con 0x7fd7f814be90 2024-03-25T18:30:27.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.075+0000 7fd7fcb15640 10 monclient(hunting): _negotiate 2024-03-25T18:30:27.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.075+0000 7fd7fcb15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:27.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.075+0000 7fd7fcb15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:27.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.075+0000 7fd7fcb15640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:30:27.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.075+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f80af940 con 0x7fd7f814be90 2024-03-25T18:30:27.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.076+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76367491 0 0) 0x7fd7f80b0d50 con 0x7fd7f80b4d60 2024-03-25T18:30:27.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.076+0000 7fd7fcb15640 10 monclient(hunting): _negotiate 2024-03-25T18:30:27.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.076+0000 7fd7fcb15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:27.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.076+0000 7fd7fcb15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:27.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.076+0000 7fd7fcb15640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:30:27.077 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.076+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f80b0a70 con 0x7fd7f80b4d60 2024-03-25T18:30:27.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401612273 0 0) 0x7fd7f80b0a70 con 0x7fd7f80b4d60 2024-03-25T18:30:27.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 >> v1:172.21.15.146:6789/0 conn(0x7fd7f814be90 legacy=0x7fd7f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 >> v1:172.21.15.102:6789/0 conn(0x7fd7f80b1730 legacy=0x7fd7f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 10 monclient: _finish_hunting 0 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 1 monclient: found mon.noname-b 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f80b1030 con 0x7fd7f80b4d60 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 10 monclient: _finish_auth 0 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 10 monclient: _check_auth_tickets 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd7f40015b0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7ffd7b640 5 monclient: authenticate success, global_id 4412 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7ffd7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7fcb15640 10 monclient: dump: 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:27.096 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:27.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.094+0000 7fd7ffd7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:27.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.100+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7f4001710 con 0x7fd7f80b4d60 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.100+0000 7fd7fcb15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.100+0000 7fd7fcb15640 1 -- 172.21.15.102:0/1217758072 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd7f4001b60 con 0x7fd7f80b4d60 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.100+0000 7fd7ffd7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.100+0000 7fd7ffd7b640 4 set_mon_vals no callback set 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.100+0000 7fd7fcb15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.100+0000 7fd7fcb15640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.100+0000 7fd7fcb15640 10 monclient: dump: 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:27.101 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:27.102 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:27.102 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:27.102 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:27.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.101+0000 7fd7ffd7b640 10 monclient: shutdown 2024-03-25T18:30:27.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.101+0000 7fd7ffd7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:27.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.101+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1217758072 >> v1:172.21.15.102:6790/0 conn(0x7fd7f80b4d60 legacy=0x7fd7f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.101+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1217758072 shutdown_connections 2024-03-25T18:30:27.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.101+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1217758072 >> 172.21.15.102:0/1217758072 conn(0x7fd7f80617a0 msgr2=0x7fd7f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.101+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1217758072 shutdown_connections 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.101+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1217758072 wait complete. 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 10 monclient: build_initial_monmap 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 10 monclient: monmap: 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 1 librados: starting msgr at 2024-03-25T18:30:27.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 1 librados: starting objecter 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 1 Processor -- start 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 1 -- start start 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 1 librados: setting wanted keys 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 1 librados: calling monclient init 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.102+0000 7fd7ffd7b640 10 monclient: init 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f80b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f80b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f814be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f81dac30 con 0x7fd7f814be90 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f81daed0 con 0x7fd7f80b1730 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f81dbb00 con 0x7fd7f80b4d60 2024-03-25T18:30:27.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:27.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7ffd7b640 10 monclient(hunting): authenticate will time out at 7658.391248s 2024-03-25T18:30:27.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7fdb17640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd7f80b1730 0x7fd7f8142e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:40848/0 (socket says 172.21.15.102:40848) 2024-03-25T18:30:27.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.103+0000 7fd7fdb17640 1 -- 172.21.15.102:0/1274654272 learned_addr learned my addr 172.21.15.102:0/1274654272 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:27.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.104+0000 7fd7ee7fc640 10 client.?.objecter ms_handle_connect 0x7fd7f80b1730 2024-03-25T18:30:27.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.104+0000 7fd7ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:27.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.105+0000 7fd7ee7fc640 10 client.?.objecter ms_handle_connect 0x7fd7f814be90 2024-03-25T18:30:27.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.105+0000 7fd7ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:27.114 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.114+0000 7fd7ee7fc640 10 client.?.objecter ms_handle_connect 0x7fd7f80b4d60 2024-03-25T18:30:27.114 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.114+0000 7fd7ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:27.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.176+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051420562 0 0) 0x7fd7f81dbb00 con 0x7fd7f80b4d60 2024-03-25T18:30:27.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.176+0000 7fd7ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:27.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.176+0000 7fd7ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:27.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.176+0000 7fd7ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:27.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.176+0000 7fd7ee7fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:30:27.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.176+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc004eb0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.183+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275161126 0 0) 0x7fd7f81dac30 con 0x7fd7f814be90 2024-03-25T18:30:27.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.183+0000 7fd7ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:27.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.183+0000 7fd7ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:27.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.183+0000 7fd7ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:27.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.183+0000 7fd7ee7fc640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:30:27.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.183+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f81dbb00 con 0x7fd7f814be90 2024-03-25T18:30:27.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.186+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1264426451 0 0) 0x7fd7dc004eb0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.187+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7f81dac30 con 0x7fd7f80b4d60 2024-03-25T18:30:27.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.187+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd7e8001950 con 0x7fd7f80b4d60 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.187+0000 7fd7ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.187+0000 7fd7ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.187+0000 7fd7ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:27.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.187+0000 7fd7ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:27.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.211+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1268172233 0 0) 0x7fd7f81dbb00 con 0x7fd7f814be90 2024-03-25T18:30:27.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.211+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7dc004eb0 con 0x7fd7f814be90 2024-03-25T18:30:27.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.211+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd7f0001b90 con 0x7fd7f814be90 2024-03-25T18:30:27.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.211+0000 7fd7ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:27.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.211+0000 7fd7ee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.211+0000 7fd7ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:27.213 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2796935437 0 0) 0x7fd7f81dac30 con 0x7fd7f80b4d60 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 >> v1:172.21.15.146:6789/0 conn(0x7fd7f80b1730 legacy=0x7fd7f8142e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 >> v1:172.21.15.102:6789/0 conn(0x7fd7f814be90 legacy=0x7fd7f813b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 1 monclient: found mon.c 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f8139a00 con 0x7fd7f80b4d60 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 5 monclient: authenticate success, global_id 4415 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 10 monclient: _renew_subs 2024-03-25T18:30:27.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:27.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f813bfb0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:27.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 10 monclient: _renew_subs 2024-03-25T18:30:27.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:27.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f8139140 con 0x7fd7f80b4d60 2024-03-25T18:30:27.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.216+0000 7fd7ffd7b640 1 librados: init done 2024-03-25T18:30:27.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.217+0000 7fd7ffd7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:27.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.217+0000 7fd7ffd7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:27.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.217+0000 7fd7ffd7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:27.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.217+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f813ad10 con 0x7fd7f80b4d60 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.521+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7e8003610 con 0x7fd7f80b4d60 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.521+0000 7fd7ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.521+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd7e8001d80 con 0x7fd7f80b4d60 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.521+0000 7fd7ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.521+0000 7fd7ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.521+0000 7fd7ee7fc640 10 monclient: dump: 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:27.522 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.523 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:27.523 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:27.523 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:27.523 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:27.523 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:27.523 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:27.523 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:27.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.521+0000 7fd7effff640 4 set_mon_vals no callback set 2024-03-25T18:30:27.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.522+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fd7e80197d0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.522+0000 7fd7fdb17640 10 monclient: get_auth_request con 0x7fd7dc06de40 auth_method 0 2024-03-25T18:30:27.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.523+0000 7fd7ee7fc640 10 client.4415.objecter ms_handle_connect 0x7fd7dc06de40 2024-03-25T18:30:27.542 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.541+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fd7e80535b0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.542 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.541+0000 7fd7ee7fc640 10 client.4415.objecter ms_dispatch 0x7fd7f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:27.542 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.541+0000 7fd7ee7fc640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:27.543 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.541+0000 7fd7ee7fc640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:27.543 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.541+0000 7fd7ee7fc640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-25T18:30:27.543 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.541+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7e804eef0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.543 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.541+0000 7fd7ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:27.543 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.541+0000 7fd7ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:27.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.785+0000 7fd7ffd7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:27.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.786+0000 7fd7ffd7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:27.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.786+0000 7fd7ffd7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:27.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.786+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7f80627b0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.821+0000 7fd7ee7fc640 1 -- 172.21.15.102:0/1274654272 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fd7e8050ff0 con 0x7fd7f80b4d60 2024-03-25T18:30:27.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.821+0000 7fd7ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:27.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.821+0000 7fd7ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:27.823 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:30:27.823 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:27.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 10 librados: watch_flush enter 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 10 librados: watch_flush exit 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 10 client.4415.objecter successfully canceled tick 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 >> v1:172.21.15.146:6800/3848252714 conn(0x7fd7dc06de40 legacy=0x7fd7dc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 10 monclient: shutdown 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 >> v1:172.21.15.102:6790/0 conn(0x7fd7f80b4d60 legacy=0x7fd7f81db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 shutdown_connections 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.823+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 >> 172.21.15.102:0/1274654272 conn(0x7fd7f80617a0 msgr2=0x7fd7f8137550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.824+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 shutdown_connections 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.824+0000 7fd7ffd7b640 1 -- 172.21.15.102:0/1274654272 wait complete. 2024-03-25T18:30:27.825 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:27.824+0000 7fd7ffd7b640 1 librados: shutdown 2024-03-25T18:30:27.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:34.644 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:34.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.755+0000 7efc7073a640 10 monclient: get_monmap_and_config 2024-03-25T18:30:34.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.755+0000 7efc7073a640 10 monclient: build_initial_monmap 2024-03-25T18:30:34.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.755+0000 7efc7073a640 10 monclient: monmap: 2024-03-25T18:30:34.756 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:34.756 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:34.756 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:30:34.755970+0000 2024-03-25T18:30:34.756 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:30:34.755970+0000 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.755+0000 7efc7073a640 1 Processor -- start 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 1 -- start start 2024-03-25T18:30:34.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 10 monclient: init 2024-03-25T18:30:34.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:34.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:34.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 10 monclient(hunting): picked mon.noname-c con 0x7efc680a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:34.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 10 monclient(hunting): picked mon.noname-b con 0x7efc6813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:34.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 10 monclient(hunting): picked mon.noname-a con 0x7efc681430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:34.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc68142520 con 0x7efc681430c0 2024-03-25T18:30:34.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680b28a0 con 0x7efc680a98f0 2024-03-25T18:30:34.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680b2a30 con 0x7efc6813f710 2024-03-25T18:30:34.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:34.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.756+0000 7efc7073a640 10 monclient(hunting): authenticate will time out at 7666.044807s 2024-03-25T18:30:34.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.767+0000 7efc6e4d6640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7efc680a98f0 0x7efc680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:57904/0 (socket says 172.21.15.102:57904) 2024-03-25T18:30:34.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.767+0000 7efc6e4d6640 1 -- 172.21.15.102:0/2707523043 learned_addr learned my addr 172.21.15.102:0/2707523043 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:34.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.778+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023502914 0 0) 0x7efc680b28a0 con 0x7efc680a98f0 2024-03-25T18:30:34.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.779+0000 7efc6d4d4640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.779+0000 7efc6d4d4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.779+0000 7efc6d4d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.779+0000 7efc6d4d4640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:30:34.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.779+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc54003520 con 0x7efc680a98f0 2024-03-25T18:30:34.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.785+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111345774 0 0) 0x7efc680b2a30 con 0x7efc6813f710 2024-03-25T18:30:34.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.785+0000 7efc6d4d4640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.785+0000 7efc6d4d4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.785+0000 7efc6d4d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.785+0000 7efc6d4d4640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:30:34.786 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.785+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc680b28a0 con 0x7efc6813f710 2024-03-25T18:30:34.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2019079796 0 0) 0x7efc54003520 con 0x7efc680a98f0 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 >> v1:172.21.15.102:6790/0 conn(0x7efc6813f710 legacy=0x7efc681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 >> v1:172.21.15.102:6789/0 conn(0x7efc681430c0 legacy=0x7efc6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 10 monclient: _finish_hunting 0 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 1 monclient: found mon.noname-c 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc680b44c0 con 0x7efc680a98f0 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 10 monclient: _finish_auth 0 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 10 monclient: _check_auth_tickets 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efc64001690 con 0x7efc680a98f0 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc7073a640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:30:34.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc7073a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc6d4d4640 10 monclient: dump: 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:34.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.793+0000 7efc7073a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:34.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.816+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc640017f0 con 0x7efc680a98f0 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.816+0000 7efc6d4d4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.816+0000 7efc6d4d4640 1 -- 172.21.15.102:0/2707523043 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efc64001c00 con 0x7efc680a98f0 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.816+0000 7efc7073a640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.816+0000 7efc7073a640 4 set_mon_vals no callback set 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.816+0000 7efc6d4d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.816+0000 7efc6d4d4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.816+0000 7efc6d4d4640 10 monclient: dump: 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.817 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:34.818 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.817+0000 7efc7073a640 10 monclient: shutdown 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.817+0000 7efc7073a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.817+0000 7efc7073a640 1 -- 172.21.15.102:0/2707523043 >> v1:172.21.15.146:6789/0 conn(0x7efc680a98f0 legacy=0x7efc680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.817+0000 7efc7073a640 1 -- 172.21.15.102:0/2707523043 shutdown_connections 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.817+0000 7efc7073a640 1 -- 172.21.15.102:0/2707523043 >> 172.21.15.102:0/2707523043 conn(0x7efc680b7ea0 msgr2=0x7efc680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.817+0000 7efc7073a640 1 -- 172.21.15.102:0/2707523043 shutdown_connections 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.817+0000 7efc7073a640 1 -- 172.21.15.102:0/2707523043 wait complete. 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 10 monclient: build_initial_monmap 2024-03-25T18:30:34.819 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 10 monclient: monmap: 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 1 librados: starting msgr at 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 1 librados: starting objecter 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 1 Processor -- start 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 1 -- start start 2024-03-25T18:30:34.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 1 librados: setting wanted keys 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 1 librados: calling monclient init 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.818+0000 7efc7073a640 10 monclient: init 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 10 monclient(hunting): picked mon.noname-a con 0x7efc680a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 10 monclient(hunting): picked mon.noname-b con 0x7efc6813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 10 monclient(hunting): picked mon.noname-c con 0x7efc681430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc68139a40 con 0x7efc680a98f0 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc68133f20 con 0x7efc6813f710 2024-03-25T18:30:34.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680ab950 con 0x7efc681430c0 2024-03-25T18:30:34.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:34.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.819+0000 7efc7073a640 10 monclient(hunting): authenticate will time out at 7666.107539s 2024-03-25T18:30:34.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.833+0000 7efc6e4d6640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efc680a98f0 0x7efc68138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33240/0 (socket says 172.21.15.102:33240) 2024-03-25T18:30:34.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.833+0000 7efc6e4d6640 1 -- 172.21.15.102:0/2125312676 learned_addr learned my addr 172.21.15.102:0/2125312676 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:34.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.835+0000 7efc5effd640 10 client.?.objecter ms_handle_connect 0x7efc680a98f0 2024-03-25T18:30:34.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.835+0000 7efc5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:34.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.836+0000 7efc5effd640 10 client.?.objecter ms_handle_connect 0x7efc681430c0 2024-03-25T18:30:34.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.836+0000 7efc5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:34.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.842+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073840717 0 0) 0x7efc68139a40 con 0x7efc680a98f0 2024-03-25T18:30:34.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.842+0000 7efc5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.842+0000 7efc5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.842+0000 7efc5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.842+0000 7efc5effd640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:30:34.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.842+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc4c004eb0 con 0x7efc680a98f0 2024-03-25T18:30:34.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.846+0000 7efc5effd640 10 client.?.objecter ms_handle_connect 0x7efc6813f710 2024-03-25T18:30:34.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.846+0000 7efc5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:34.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.849+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443684477 0 0) 0x7efc680ab950 con 0x7efc681430c0 2024-03-25T18:30:34.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.849+0000 7efc5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.849+0000 7efc5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.849+0000 7efc5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.849+0000 7efc5effd640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:30:34.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.849+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc68139a40 con 0x7efc681430c0 2024-03-25T18:30:34.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.856+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834595462 0 0) 0x7efc68133f20 con 0x7efc6813f710 2024-03-25T18:30:34.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.856+0000 7efc5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:34.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.856+0000 7efc5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:34.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.856+0000 7efc5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:34.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.856+0000 7efc5effd640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:30:34.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.856+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc680ab950 con 0x7efc6813f710 2024-03-25T18:30:34.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.872+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1543135244 0 0) 0x7efc4c004eb0 con 0x7efc680a98f0 2024-03-25T18:30:34.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.872+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc68133f20 con 0x7efc680a98f0 2024-03-25T18:30:34.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.872+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efc64001bc0 con 0x7efc680a98f0 2024-03-25T18:30:34.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.872+0000 7efc5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.872+0000 7efc5effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:34.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.872+0000 7efc5effd640 10 monclient(hunting): dump: 2024-03-25T18:30:34.873 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:34.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.872+0000 7efc5effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.876+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3549450544 0 0) 0x7efc68139a40 con 0x7efc681430c0 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.876+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc4c004eb0 con 0x7efc681430c0 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.876+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efc60001b10 con 0x7efc681430c0 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.876+0000 7efc5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.876+0000 7efc5effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.876+0000 7efc5effd640 10 monclient(hunting): dump: 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:34.877 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.878+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 224433319 0 0) 0x7efc680ab950 con 0x7efc6813f710 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.878+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc68139a40 con 0x7efc6813f710 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.878+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efc58001930 con 0x7efc6813f710 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.878+0000 7efc5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.878+0000 7efc5effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.878+0000 7efc5effd640 10 monclient(hunting): dump: 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:34.879 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:34.880 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:34.892 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 543767358 0 0) 0x7efc4c004eb0 con 0x7efc681430c0 2024-03-25T18:30:34.892 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 >> v1:172.21.15.146:6789/0 conn(0x7efc6813f710 legacy=0x7efc68133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.892 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 >> v1:172.21.15.102:6789/0 conn(0x7efc680a98f0 legacy=0x7efc68138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:34.892 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:34.892 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 1 monclient: found mon.c 2024-03-25T18:30:34.892 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc680abc30 con 0x7efc681430c0 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 10 monclient: _finish_auth 0 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 5 monclient: authenticate success, global_id 4421 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 10 monclient: _renew_subs 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc680abf10 con 0x7efc681430c0 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 10 monclient: _renew_subs 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc68139370 con 0x7efc681430c0 2024-03-25T18:30:34.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.891+0000 7efc7073a640 1 librados: init done 2024-03-25T18:30:34.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.892+0000 7efc7073a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:34.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.892+0000 7efc7073a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:34.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.892+0000 7efc7073a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:34.894 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:34.892+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc680627b0 con 0x7efc681430c0 2024-03-25T18:30:35.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.208+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc60002000 con 0x7efc681430c0 2024-03-25T18:30:35.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.208+0000 7efc5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:35.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.208+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efc60003b10 con 0x7efc681430c0 2024-03-25T18:30:35.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.208+0000 7efc5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:35.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.208+0000 7efc5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.208+0000 7efc5effd640 10 monclient: dump: 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:35.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.208+0000 7efc6ccd3640 4 set_mon_vals no callback set 2024-03-25T18:30:35.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.209+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7efc60003d30 con 0x7efc681430c0 2024-03-25T18:30:35.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.209+0000 7efc6e4d6640 10 monclient: get_auth_request con 0x7efc4c06e160 auth_method 0 2024-03-25T18:30:35.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.210+0000 7efc5effd640 10 client.4421.objecter ms_handle_connect 0x7efc4c06e160 2024-03-25T18:30:35.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.215+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7efc600537f0 con 0x7efc681430c0 2024-03-25T18:30:35.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.215+0000 7efc5effd640 10 client.4421.objecter ms_dispatch 0x7efc68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:35.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.215+0000 7efc5effd640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:35.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.215+0000 7efc5effd640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:35.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.215+0000 7efc5effd640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-25T18:30:35.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.215+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc6004f300 con 0x7efc681430c0 2024-03-25T18:30:35.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.215+0000 7efc5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:35.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.215+0000 7efc5effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:35.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.437+0000 7efc7073a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:35.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.437+0000 7efc7073a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:35.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.437+0000 7efc7073a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:35.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.437+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc68138980 con 0x7efc681430c0 2024-03-25T18:30:35.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.474+0000 7efc5effd640 1 -- 172.21.15.102:0/2125312676 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7efc60051230 con 0x7efc681430c0 2024-03-25T18:30:35.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.475+0000 7efc5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:35.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.475+0000 7efc5effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:35.476 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:30:35.476 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.477+0000 7efc7073a640 10 librados: watch_flush enter 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.477+0000 7efc7073a640 10 librados: watch_flush exit 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.477+0000 7efc7073a640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.477+0000 7efc7073a640 10 client.4421.objecter successfully canceled tick 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.477+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 >> v1:172.21.15.146:6800/3848252714 conn(0x7efc4c06e160 legacy=0x7efc4c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.477+0000 7efc7073a640 10 monclient: shutdown 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.477+0000 7efc7073a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.477+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 >> v1:172.21.15.102:6790/0 conn(0x7efc681430c0 legacy=0x7efc68133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.478+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 shutdown_connections 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.478+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 >> 172.21.15.102:0/2125312676 conn(0x7efc680b7ea0 msgr2=0x7efc68074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:35.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.478+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 shutdown_connections 2024-03-25T18:30:35.480 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.478+0000 7efc7073a640 1 -- 172.21.15.102:0/2125312676 wait complete. 2024-03-25T18:30:35.480 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:35.478+0000 7efc7073a640 1 librados: shutdown 2024-03-25T18:30:35.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:42.298 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.415+0000 7f1587325640 10 monclient: get_monmap_and_config 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.415+0000 7f1587325640 10 monclient: build_initial_monmap 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.415+0000 7f1587325640 10 monclient: monmap: 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:30:42.416973+0000 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:30:42.416973+0000 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:42.417 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:30:42.418 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:30:42.418 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:42.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.416+0000 7f1587325640 1 Processor -- start 2024-03-25T18:30:42.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.416+0000 7f1587325640 1 -- start start 2024-03-25T18:30:42.418 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.416+0000 7f1587325640 10 monclient: init 2024-03-25T18:30:42.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:42.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:42.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 10 monclient(hunting): picked mon.noname-b con 0x7f15800a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:42.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 10 monclient(hunting): picked mon.noname-a con 0x7f158013f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:42.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 10 monclient(hunting): picked mon.noname-c con 0x7f15801430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:42.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1580142520 con 0x7f158013f710 2024-03-25T18:30:42.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15800b28a0 con 0x7f15801430c0 2024-03-25T18:30:42.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15800b2a30 con 0x7f15800a98f0 2024-03-25T18:30:42.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:42.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.417+0000 7f1587325640 10 monclient(hunting): authenticate will time out at 7673.706925s 2024-03-25T18:30:42.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.431+0000 7f15858c2640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f15801430c0 0x7f158014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:54332/0 (socket says 172.21.15.102:54332) 2024-03-25T18:30:42.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.431+0000 7f15858c2640 1 -- 172.21.15.102:0/3182109896 learned_addr learned my addr 172.21.15.102:0/3182109896 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:42.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.437+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929445116 0 0) 0x7f15800b28a0 con 0x7f15801430c0 2024-03-25T18:30:42.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.437+0000 7f1577fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.437+0000 7f1577fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.437+0000 7f1577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.437+0000 7f1577fff640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:30:42.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.437+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c003520 con 0x7f15801430c0 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.450+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601083622 0 0) 0x7f1580142520 con 0x7f158013f710 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.450+0000 7f1577fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.450+0000 7f1577fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.450+0000 7f1577fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.450+0000 7f1577fff640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.450+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15800b28a0 con 0x7f158013f710 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3619904742 0 0) 0x7f156c003520 con 0x7f15801430c0 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 >> v1:172.21.15.102:6790/0 conn(0x7f15800a98f0 legacy=0x7f15800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 >> v1:172.21.15.102:6789/0 conn(0x7f158013f710 legacy=0x7f15801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 1 monclient: found mon.noname-c 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:42.453 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15800b44c0 con 0x7f15801430c0 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 10 monclient: _finish_auth 0 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1578001a10 con 0x7f15801430c0 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1587325640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1587325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1577fff640 10 monclient: dump: 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.454 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.455 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:42.455 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:42.455 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:42.455 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:42.455 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:42.455 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.451+0000 7f1587325640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.458+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1578001bf0 con 0x7f15801430c0 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.458+0000 7f1577fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.458+0000 7f1577fff640 1 -- 172.21.15.102:0/3182109896 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1578002000 con 0x7f15801430c0 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.458+0000 7f1577fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.458+0000 7f1577fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.458+0000 7f1577fff640 10 monclient: dump: 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:42.460 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:42.461 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:42.461 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:42.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.458+0000 7f1587325640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:42.461 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.458+0000 7f1587325640 4 set_mon_vals no callback set 2024-03-25T18:30:42.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.459+0000 7f1587325640 10 monclient: shutdown 2024-03-25T18:30:42.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.459+0000 7f1587325640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:42.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.459+0000 7f1587325640 1 -- 172.21.15.102:0/3182109896 >> v1:172.21.15.146:6789/0 conn(0x7f15801430c0 legacy=0x7f158014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.459+0000 7f1587325640 1 -- 172.21.15.102:0/3182109896 shutdown_connections 2024-03-25T18:30:42.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.459+0000 7f1587325640 1 -- 172.21.15.102:0/3182109896 >> 172.21.15.102:0/3182109896 conn(0x7f15800b7ea0 msgr2=0x7f15800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:42.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.459+0000 7f1587325640 1 -- 172.21.15.102:0/3182109896 shutdown_connections 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.459+0000 7f1587325640 1 -- 172.21.15.102:0/3182109896 wait complete. 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 10 monclient: build_initial_monmap 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 10 monclient: monmap: 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 1 librados: starting msgr at 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 1 librados: starting objecter 2024-03-25T18:30:42.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 1 Processor -- start 2024-03-25T18:30:42.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 1 -- start start 2024-03-25T18:30:42.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 1 librados: setting wanted keys 2024-03-25T18:30:42.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 1 librados: calling monclient init 2024-03-25T18:30:42.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.460+0000 7f1587325640 10 monclient: init 2024-03-25T18:30:42.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:42.464 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:42.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 10 monclient(hunting): picked mon.noname-a con 0x7f15800a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:42.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 10 monclient(hunting): picked mon.noname-c con 0x7f158013f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:42.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 10 monclient(hunting): picked mon.noname-b con 0x7f15801430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:42.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15801dacc0 con 0x7f15800a98f0 2024-03-25T18:30:42.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1580139990 con 0x7f15801430c0 2024-03-25T18:30:42.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15801db860 con 0x7f158013f710 2024-03-25T18:30:42.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:42.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.461+0000 7f1587325640 10 monclient(hunting): authenticate will time out at 7673.750387s 2024-03-25T18:30:42.483 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.482+0000 7f15850c1640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f15800a98f0 0x7f1580137a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51724/0 (socket says 172.21.15.102:51724) 2024-03-25T18:30:42.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.482+0000 7f15850c1640 1 -- 172.21.15.102:0/4113344151 learned_addr learned my addr 172.21.15.102:0/4113344151 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:42.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.483+0000 7f1575ffb640 10 client.?.objecter ms_handle_connect 0x7f15800a98f0 2024-03-25T18:30:42.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.483+0000 7f1575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:42.485 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.483+0000 7f1575ffb640 10 client.?.objecter ms_handle_connect 0x7f15801430c0 2024-03-25T18:30:42.485 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.483+0000 7f1575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:42.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.486+0000 7f1575ffb640 10 client.?.objecter ms_handle_connect 0x7f158013f710 2024-03-25T18:30:42.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.486+0000 7f1575ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:42.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.497+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879298169 0 0) 0x7f15801dacc0 con 0x7f15800a98f0 2024-03-25T18:30:42.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.497+0000 7f1575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.497+0000 7f1575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.497+0000 7f1575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.497+0000 7f1575ffb640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:30:42.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.497+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1564004eb0 con 0x7f15800a98f0 2024-03-25T18:30:42.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481698121 0 0) 0x7f15801db860 con 0x7f158013f710 2024-03-25T18:30:42.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:30:42.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15801dacc0 con 0x7f158013f710 2024-03-25T18:30:42.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960267813 0 0) 0x7f1580139990 con 0x7f15801430c0 2024-03-25T18:30:42.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:30:42.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:42.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:42.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:30:42.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.498+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15801db860 con 0x7f15801430c0 2024-03-25T18:30:42.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.520+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2951513435 0 0) 0x7f15801db860 con 0x7f15801430c0 2024-03-25T18:30:42.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.520+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1580139990 con 0x7f15801430c0 2024-03-25T18:30:42.522 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.520+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1578001f00 con 0x7f15801430c0 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.520+0000 7f1575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.520+0000 7f1575ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.520+0000 7f1575ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.520+0000 7f1575ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:42.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.521+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3966397279 0 0) 0x7f1564004eb0 con 0x7f15800a98f0 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.521+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15801db860 con 0x7f15800a98f0 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.521+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f157c0019d0 con 0x7f15800a98f0 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.521+0000 7f1575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.521+0000 7f1575ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.521+0000 7f1575ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:42.524 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.523+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1736467974 0 0) 0x7f15801dacc0 con 0x7f158013f710 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.523+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1564004eb0 con 0x7f158013f710 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.523+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1570001930 con 0x7f158013f710 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.523+0000 7f1575ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.523+0000 7f1575ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.523+0000 7f1575ffb640 10 monclient(hunting): dump: 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.525 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.526 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.526 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:42.526 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:42.526 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:42.526 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:42.526 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:42.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2796817236 0 0) 0x7f15801db860 con 0x7f15800a98f0 2024-03-25T18:30:42.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 >> v1:172.21.15.102:6790/0 conn(0x7f158013f710 legacy=0x7f15801db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 >> v1:172.21.15.146:6789/0 conn(0x7f15801430c0 legacy=0x7f15801e2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 1 monclient: found mon.a 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1580135040 con 0x7f15800a98f0 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 10 monclient: _finish_auth 0 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1575ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1587325640 5 monclient: authenticate success, global_id 4446 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1587325640 10 monclient: _renew_subs 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1587325640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.535+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15801352f0 con 0x7f15800a98f0 2024-03-25T18:30:42.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.536+0000 7f1587325640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:42.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.536+0000 7f1587325640 10 monclient: _renew_subs 2024-03-25T18:30:42.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.536+0000 7f1587325640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:42.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.536+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1580138a70 con 0x7f15800a98f0 2024-03-25T18:30:42.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.536+0000 7f1587325640 1 librados: init done 2024-03-25T18:30:42.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.537+0000 7f1587325640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:42.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.537+0000 7f1587325640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:42.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.537+0000 7f1587325640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:42.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.537+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1580138d20 con 0x7f15800a98f0 2024-03-25T18:30:42.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.848+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f157c003610 con 0x7f15800a98f0 2024-03-25T18:30:42.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.848+0000 7f1575ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:42.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.848+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f157c003a20 con 0x7f15800a98f0 2024-03-25T18:30:42.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.848+0000 7f1575ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:42.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.848+0000 7f1575ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:42.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.848+0000 7f1575ffb640 10 monclient: dump: 2024-03-25T18:30:42.850 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:42.850 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:42.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.848+0000 7f15777fe640 4 set_mon_vals no callback set 2024-03-25T18:30:42.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.849+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f157c003c40 con 0x7f15800a98f0 2024-03-25T18:30:42.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.849+0000 7f15848c0640 10 monclient: get_auth_request con 0x7f156406e0b0 auth_method 0 2024-03-25T18:30:42.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.850+0000 7f1575ffb640 10 client.4446.objecter ms_handle_connect 0x7f156406e0b0 2024-03-25T18:30:42.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.854+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f157c054820 con 0x7f15800a98f0 2024-03-25T18:30:42.856 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.854+0000 7f1575ffb640 10 client.4446.objecter ms_dispatch 0x7f1580000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:42.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.854+0000 7f1575ffb640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:42.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.854+0000 7f1575ffb640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:42.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.854+0000 7f1575ffb640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-25T18:30:42.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.854+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f157c050330 con 0x7f15800a98f0 2024-03-25T18:30:42.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.855+0000 7f1575ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:42.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:42.855+0000 7f1575ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:43.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.108+0000 7f1587325640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:43.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.109+0000 7f1587325640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:43.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.109+0000 7f1587325640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:30:43.111 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.109+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15801daa20 con 0x7f15800a98f0 2024-03-25T18:30:43.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.159+0000 7f1575ffb640 1 -- 172.21.15.102:0/4113344151 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f157c052260 con 0x7f15800a98f0 2024-03-25T18:30:43.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.159+0000 7f1575ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:43.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.159+0000 7f1575ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:43.161 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:30:43.162 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:43.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 10 librados: watch_flush enter 2024-03-25T18:30:43.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 10 librados: watch_flush exit 2024-03-25T18:30:43.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-25T18:30:43.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 10 client.4446.objecter successfully canceled tick 2024-03-25T18:30:43.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 >> v1:172.21.15.146:6800/3848252714 conn(0x7f156406e0b0 legacy=0x7f1564070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:43.165 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 10 monclient: shutdown 2024-03-25T18:30:43.165 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:43.165 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 >> v1:172.21.15.102:6789/0 conn(0x7f15800a98f0 legacy=0x7f1580137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:43.165 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 shutdown_connections 2024-03-25T18:30:43.165 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 >> 172.21.15.102:0/4113344151 conn(0x7f15800b7ea0 msgr2=0x7f1580074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:43.165 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 shutdown_connections 2024-03-25T18:30:43.165 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 1 -- 172.21.15.102:0/4113344151 wait complete. 2024-03-25T18:30:43.165 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:43.162+0000 7f1587325640 1 librados: shutdown 2024-03-25T18:30:43.176 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:43.692 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:04:59.928 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:43.695 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:04:59.930 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:43.706 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:04:59.941 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:43.731 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:04:59.967 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:43.753 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:04:59.966 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:43.758 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:04:59.971 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:43.758 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:04:59.994 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:43.785 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:04:59.998 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:43.789 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:05:00.025 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:49.983 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:50.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.093+0000 7fb3b9425640 10 monclient: get_monmap_and_config 2024-03-25T18:30:50.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.093+0000 7fb3b9425640 10 monclient: build_initial_monmap 2024-03-25T18:30:50.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.093+0000 7fb3b9425640 10 monclient: monmap: 2024-03-25T18:30:50.095 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:50.095 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:50.095 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:30:50.095450+0000 2024-03-25T18:30:50.096 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:30:50.095450+0000 2024-03-25T18:30:50.096 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:50.096 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:50.096 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:50.096 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:30:50.096 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:30:50.096 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:50.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.094+0000 7fb3b9425640 1 Processor -- start 2024-03-25T18:30:50.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.094+0000 7fb3b9425640 1 -- start start 2024-03-25T18:30:50.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.094+0000 7fb3b9425640 10 monclient: init 2024-03-25T18:30:50.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b40a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b41430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b4142520 con 0x7fb3b413f710 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b40b28a0 con 0x7fb3b40a98f0 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b40b2a30 con 0x7fb3b41430c0 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:50.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.095+0000 7fb3b9425640 10 monclient(hunting): authenticate will time out at 7681.384197s 2024-03-25T18:30:50.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.103+0000 7fb3b37fe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fb3b41430c0 0x7fb3b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:37100/0 (socket says 172.21.15.102:37100) 2024-03-25T18:30:50.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.103+0000 7fb3b37fe640 1 -- 172.21.15.102:0/2244157629 learned_addr learned my addr 172.21.15.102:0/2244157629 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:50.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.123+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732054582 0 0) 0x7fb3b40b2a30 con 0x7fb3b41430c0 2024-03-25T18:30:50.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.123+0000 7fb3b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:50.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.123+0000 7fb3b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:50.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.123+0000 7fb3b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:50.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.123+0000 7fb3b27fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:30:50.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.123+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a0003520 con 0x7fb3b41430c0 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339232697 0 0) 0x7fb3b40b28a0 con 0x7fb3b40a98f0 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b40b2a30 con 0x7fb3b40a98f0 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079740091 0 0) 0x7fb3b4142520 con 0x7fb3b413f710 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:50.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:50.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:30:50.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.133+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b40b28a0 con 0x7fb3b413f710 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.139+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850646573 0 0) 0x7fb3a0003520 con 0x7fb3b41430c0 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 >> v1:172.21.15.146:6789/0 conn(0x7fb3b40a98f0 legacy=0x7fb3b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 >> v1:172.21.15.102:6789/0 conn(0x7fb3b413f710 legacy=0x7fb3b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 1 monclient: found mon.noname-b 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b40b44c0 con 0x7fb3b41430c0 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:50.142 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb3ac001a10 con 0x7fb3b41430c0 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b27fc640 10 monclient: dump: 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b9425640 5 monclient: authenticate success, global_id 4430 2024-03-25T18:30:50.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.140+0000 7fb3b9425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:50.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.144+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3ac001b70 con 0x7fb3b41430c0 2024-03-25T18:30:50.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.144+0000 7fb3b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:50.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.144+0000 7fb3b27fc640 1 -- 172.21.15.102:0/2244157629 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb3ac001f80 con 0x7fb3b41430c0 2024-03-25T18:30:50.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.144+0000 7fb3b9425640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:50.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.144+0000 7fb3b9425640 4 set_mon_vals no callback set 2024-03-25T18:30:50.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.144+0000 7fb3b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.144+0000 7fb3b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.144+0000 7fb3b27fc640 10 monclient: dump: 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:50.147 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:50.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.145+0000 7fb3b9425640 10 monclient: shutdown 2024-03-25T18:30:50.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.145+0000 7fb3b9425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.145+0000 7fb3b9425640 1 -- 172.21.15.102:0/2244157629 >> v1:172.21.15.102:6790/0 conn(0x7fb3b41430c0 legacy=0x7fb3b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.145+0000 7fb3b9425640 1 -- 172.21.15.102:0/2244157629 shutdown_connections 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.145+0000 7fb3b9425640 1 -- 172.21.15.102:0/2244157629 >> 172.21.15.102:0/2244157629 conn(0x7fb3b40b7ea0 msgr2=0x7fb3b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.145+0000 7fb3b9425640 1 -- 172.21.15.102:0/2244157629 shutdown_connections 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.145+0000 7fb3b9425640 1 -- 172.21.15.102:0/2244157629 wait complete. 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.146+0000 7fb3b9425640 10 monclient: build_initial_monmap 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.146+0000 7fb3b9425640 10 monclient: monmap: 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:30:50.150 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.146+0000 7fb3b9425640 1 librados: starting msgr at 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.146+0000 7fb3b9425640 1 librados: starting objecter 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.146+0000 7fb3b9425640 1 Processor -- start 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 1 -- start start 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 1 librados: setting wanted keys 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 1 librados: calling monclient init 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 10 monclient: init 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b40a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:50.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b413f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:50.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b41430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:50.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b4138b40 con 0x7fb3b41430c0 2024-03-25T18:30:50.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b4138e20 con 0x7fb3b40a98f0 2024-03-25T18:30:50.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b4139100 con 0x7fb3b413f710 2024-03-25T18:30:50.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:50.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.147+0000 7fb3b9425640 10 monclient(hunting): authenticate will time out at 7681.436824s 2024-03-25T18:30:50.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.160+0000 7fb3b2ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb3b40a98f0 0x7fb3b4137380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:58712/0 (socket says 172.21.15.102:58712) 2024-03-25T18:30:50.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.160+0000 7fb3b2ffd640 1 -- 172.21.15.102:0/1682241865 learned_addr learned my addr 172.21.15.102:0/1682241865 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:50.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.160+0000 7fb3ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb3b40a98f0 2024-03-25T18:30:50.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.160+0000 7fb3ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:50.167 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.166+0000 7fb3ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb3b41430c0 2024-03-25T18:30:50.168 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.166+0000 7fb3ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:50.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.175+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192887268 0 0) 0x7fb3b4138b40 con 0x7fb3b41430c0 2024-03-25T18:30:50.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.175+0000 7fb3ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:50.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.175+0000 7fb3ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:50.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.175+0000 7fb3ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:50.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.175+0000 7fb3ab7fe640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:30:50.177 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.175+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb394004eb0 con 0x7fb3b41430c0 2024-03-25T18:30:50.186 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.185+0000 7fb3ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb3b413f710 2024-03-25T18:30:50.186 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.185+0000 7fb3ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:50.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.195+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302319608 0 0) 0x7fb3b4139100 con 0x7fb3b413f710 2024-03-25T18:30:50.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.195+0000 7fb3ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:50.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.195+0000 7fb3ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:50.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.195+0000 7fb3ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:50.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.195+0000 7fb3ab7fe640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:30:50.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.195+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b4138b40 con 0x7fb3b413f710 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.197+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 210282585 0 0) 0x7fb394004eb0 con 0x7fb3b41430c0 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.197+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b4139100 con 0x7fb3b41430c0 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.197+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb3ac001e80 con 0x7fb3b41430c0 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.197+0000 7fb3ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.197+0000 7fb3ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.197+0000 7fb3ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:50.199 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:50.200 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:50.200 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:50.200 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:50.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.197+0000 7fb3ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:50.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.209+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529176737 0 0) 0x7fb3b4138b40 con 0x7fb3b413f710 2024-03-25T18:30:50.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.209+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb394004eb0 con 0x7fb3b413f710 2024-03-25T18:30:50.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.210+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb39c001870 con 0x7fb3b413f710 2024-03-25T18:30:50.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.210+0000 7fb3ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.210+0000 7fb3ab7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.210+0000 7fb3ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:50.212 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3475805107 0 0) 0x7fb394004eb0 con 0x7fb3b413f710 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 >> v1:172.21.15.146:6789/0 conn(0x7fb3b40a98f0 legacy=0x7fb3b4137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 >> v1:172.21.15.102:6789/0 conn(0x7fb3b41430c0 legacy=0x7fb3b41e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 1 monclient: found mon.c 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b41397a0 con 0x7fb3b413f710 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:50.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 5 monclient: authenticate success, global_id 4433 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 10 monclient: _renew_subs 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b4136aa0 con 0x7fb3b413f710 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 10 monclient: _renew_subs 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b413c320 con 0x7fb3b413f710 2024-03-25T18:30:50.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.224+0000 7fb3b9425640 1 librados: init done 2024-03-25T18:30:50.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.225+0000 7fb3b9425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:50.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.225+0000 7fb3b9425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:50.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.225+0000 7fb3b9425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:50.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.225+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b41395b0 con 0x7fb3b413f710 2024-03-25T18:30:50.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.442+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb39c003610 con 0x7fb3b413f710 2024-03-25T18:30:50.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.442+0000 7fb3ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:50.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.442+0000 7fb3b1ffb640 4 set_mon_vals no callback set 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.467+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb39c001d80 con 0x7fb3b413f710 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.467+0000 7fb3ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.467+0000 7fb3ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.467+0000 7fb3ab7fe640 10 monclient: dump: 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:50.469 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:50.470 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:50.470 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:50.470 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:50.470 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:50.470 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:50.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.468+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fb39c003ae0 con 0x7fb3b413f710 2024-03-25T18:30:50.471 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.469+0000 7fb3b2ffd640 10 monclient: get_auth_request con 0x7fb39406df20 auth_method 0 2024-03-25T18:30:50.471 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.469+0000 7fb3ab7fe640 10 client.4433.objecter ms_handle_connect 0x7fb39406df20 2024-03-25T18:30:50.474 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.472+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fb39c053610 con 0x7fb3b413f710 2024-03-25T18:30:50.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.472+0000 7fb3ab7fe640 10 client.4433.objecter ms_dispatch 0x7fb3b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:50.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.472+0000 7fb3ab7fe640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:50.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.472+0000 7fb3ab7fe640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:50.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.472+0000 7fb3ab7fe640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-25T18:30:50.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.472+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb39c04f120 con 0x7fb3b413f710 2024-03-25T18:30:50.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.472+0000 7fb3ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:50.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.472+0000 7fb3ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:50.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.680+0000 7fb3b9425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:50.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.680+0000 7fb3b9425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:50.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.680+0000 7fb3b9425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:30:50.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.680+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3b40627b0 con 0x7fb3b413f710 2024-03-25T18:30:50.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.706+0000 7fb3ab7fe640 1 -- 172.21.15.102:0/1682241865 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fb39c051050 con 0x7fb3b413f710 2024-03-25T18:30:50.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.706+0000 7fb3ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:50.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.706+0000 7fb3ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:50.708 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:30:50.708 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:50.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 10 librados: watch_flush enter 2024-03-25T18:30:50.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 10 librados: watch_flush exit 2024-03-25T18:30:50.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 10 client.4433.objecter successfully canceled tick 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 >> v1:172.21.15.146:6800/3848252714 conn(0x7fb39406df20 legacy=0x7fb3940703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 10 monclient: shutdown 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 >> v1:172.21.15.102:6790/0 conn(0x7fb3b413f710 legacy=0x7fb3b4137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 shutdown_connections 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 >> 172.21.15.102:0/1682241865 conn(0x7fb3b40b7ea0 msgr2=0x7fb3b414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 shutdown_connections 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 1 -- 172.21.15.102:0/1682241865 wait complete. 2024-03-25T18:30:50.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:50.708+0000 7fb3b9425640 1 librados: shutdown 2024-03-25T18:30:50.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:57.530 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:57.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.641+0000 7f75003f4640 10 monclient: get_monmap_and_config 2024-03-25T18:30:57.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.641+0000 7f75003f4640 10 monclient: build_initial_monmap 2024-03-25T18:30:57.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.641+0000 7f75003f4640 10 monclient: monmap: 2024-03-25T18:30:57.643 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:57.643 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:57.643 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:30:57.642606+0000 2024-03-25T18:30:57.643 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:30:57.642606+0000 2024-03-25T18:30:57.643 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:57.644 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:57.644 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:57.644 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:30:57.644 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:30:57.644 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:57.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.642+0000 7f75003f4640 1 Processor -- start 2024-03-25T18:30:57.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.642+0000 7f75003f4640 1 -- start start 2024-03-25T18:30:57.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.642+0000 7f75003f4640 10 monclient: init 2024-03-25T18:30:57.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:57.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:57.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 10 monclient(hunting): picked mon.noname-b con 0x7f74f80a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:57.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 10 monclient(hunting): picked mon.noname-a con 0x7f74f813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:57.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 10 monclient(hunting): picked mon.noname-c con 0x7f74f81430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:57.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8142520 con 0x7f74f813f710 2024-03-25T18:30:57.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f80b28a0 con 0x7f74f81430c0 2024-03-25T18:30:57.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f80b2a30 con 0x7f74f80a98f0 2024-03-25T18:30:57.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:57.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.643+0000 7f75003f4640 10 monclient(hunting): authenticate will time out at 7688.931536s 2024-03-25T18:30:57.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.655+0000 7f74fe991640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f74f81430c0 0x7f74f814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:60934/0 (socket says 172.21.15.102:60934) 2024-03-25T18:30:57.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.655+0000 7f74fe991640 1 -- 172.21.15.102:0/508716736 learned_addr learned my addr 172.21.15.102:0/508716736 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:57.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.663+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719976389 0 0) 0x7f74f80b2a30 con 0x7f74f80a98f0 2024-03-25T18:30:57.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.663+0000 7f74fd18e640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.663+0000 7f74fd18e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.663+0000 7f74fd18e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.663+0000 7f74fd18e640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:30:57.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.663+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e4003520 con 0x7f74f80a98f0 2024-03-25T18:30:57.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.671+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609100071 0 0) 0x7f74f80b28a0 con 0x7f74f81430c0 2024-03-25T18:30:57.672 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.671+0000 7f74fd18e640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.672 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.671+0000 7f74fd18e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.672 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.671+0000 7f74fd18e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.672 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.671+0000 7f74fd18e640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:30:57.672 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.671+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f80b2a30 con 0x7f74f81430c0 2024-03-25T18:30:57.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.679+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027709646 0 0) 0x7f74f8142520 con 0x7f74f813f710 2024-03-25T18:30:57.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.679+0000 7f74fd18e640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.679+0000 7f74fd18e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.679+0000 7f74fd18e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.679+0000 7f74fd18e640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:30:57.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.679+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f80b28a0 con 0x7f74f813f710 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845164337 0 0) 0x7f74e4003520 con 0x7f74f80a98f0 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 >> v1:172.21.15.146:6789/0 conn(0x7f74f81430c0 legacy=0x7f74f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 >> v1:172.21.15.102:6789/0 conn(0x7f74f813f710 legacy=0x7f74f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 10 monclient: _finish_hunting 0 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 1 monclient: found mon.noname-b 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f80b44c0 con 0x7f74f80a98f0 2024-03-25T18:30:57.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 10 monclient: _finish_auth 0 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 10 monclient: _check_auth_tickets 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f74f4001690 con 0x7f74f80a98f0 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f74fd18e640 10 monclient: dump: 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:57.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f75003f4640 5 monclient: authenticate success, global_id 4436 2024-03-25T18:30:57.684 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.681+0000 7f75003f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:57.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.692+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74f40017f0 con 0x7f74f80a98f0 2024-03-25T18:30:57.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.692+0000 7f74fd18e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:57.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.692+0000 7f74fd18e640 1 -- 172.21.15.102:0/508716736 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f74f4001c00 con 0x7f74f80a98f0 2024-03-25T18:30:57.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.692+0000 7f74fd18e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.692+0000 7f74fd18e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.692+0000 7f74fd18e640 10 monclient: dump: 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.692+0000 7f75003f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:57.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.692+0000 7f75003f4640 4 set_mon_vals no callback set 2024-03-25T18:30:57.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.693+0000 7f75003f4640 10 monclient: shutdown 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.693+0000 7f75003f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.693+0000 7f75003f4640 1 -- 172.21.15.102:0/508716736 >> v1:172.21.15.102:6790/0 conn(0x7f74f80a98f0 legacy=0x7f74f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.693+0000 7f75003f4640 1 -- 172.21.15.102:0/508716736 shutdown_connections 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.693+0000 7f75003f4640 1 -- 172.21.15.102:0/508716736 >> 172.21.15.102:0/508716736 conn(0x7f74f80b7ea0 msgr2=0x7f74f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.693+0000 7f75003f4640 1 -- 172.21.15.102:0/508716736 shutdown_connections 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.693+0000 7f75003f4640 1 -- 172.21.15.102:0/508716736 wait complete. 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 10 monclient: build_initial_monmap 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 10 monclient: monmap: 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:30:57.696 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 1 librados: starting msgr at 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 1 librados: starting objecter 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 1 Processor -- start 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 1 -- start start 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 1 librados: setting wanted keys 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 1 librados: calling monclient init 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.694+0000 7f75003f4640 10 monclient: init 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 10 monclient(hunting): picked mon.noname-b con 0x7f74f80a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 10 monclient(hunting): picked mon.noname-a con 0x7f74f813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 10 monclient(hunting): picked mon.noname-c con 0x7f74f81430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:30:57.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8135610 con 0x7f74f813f710 2024-03-25T18:30:57.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8137230 con 0x7f74f80a98f0 2024-03-25T18:30:57.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8135cf0 con 0x7f74f81430c0 2024-03-25T18:30:57.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:57.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.695+0000 7f75003f4640 10 monclient(hunting): authenticate will time out at 7688.983124s 2024-03-25T18:30:57.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.708+0000 7f74fd98f640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f74f813f710 0x7f74f81dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54302/0 (socket says 172.21.15.102:54302) 2024-03-25T18:30:57.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.708+0000 7f74fd98f640 1 -- 172.21.15.102:0/3506435250 learned_addr learned my addr 172.21.15.102:0/3506435250 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:30:57.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.710+0000 7f74eeffd640 10 client.?.objecter ms_handle_connect 0x7f74f80a98f0 2024-03-25T18:30:57.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.710+0000 7f74eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:57.716 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.715+0000 7f74eeffd640 10 client.?.objecter ms_handle_connect 0x7f74f81430c0 2024-03-25T18:30:57.716 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.715+0000 7f74eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:57.716 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.715+0000 7f74eeffd640 10 client.?.objecter ms_handle_connect 0x7f74f813f710 2024-03-25T18:30:57.716 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.715+0000 7f74eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:57.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.723+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599700839 0 0) 0x7f74f8137230 con 0x7f74f80a98f0 2024-03-25T18:30:57.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.723+0000 7f74eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.723+0000 7f74eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.723+0000 7f74eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.723+0000 7f74eeffd640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:30:57.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.723+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc004eb0 con 0x7f74f80a98f0 2024-03-25T18:30:57.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.734+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208813914 0 0) 0x7f74f8135cf0 con 0x7f74f81430c0 2024-03-25T18:30:57.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.734+0000 7f74eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.734+0000 7f74eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.734+0000 7f74eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.734+0000 7f74eeffd640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:30:57.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.734+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f8137230 con 0x7f74f81430c0 2024-03-25T18:30:57.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.736+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580626088 0 0) 0x7f74f8135610 con 0x7f74f813f710 2024-03-25T18:30:57.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.736+0000 7f74eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:57.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.736+0000 7f74eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:57.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.736+0000 7f74eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:57.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.736+0000 7f74eeffd640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:30:57.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.736+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f8135cf0 con 0x7f74f813f710 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.740+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1739810938 0 0) 0x7f74dc004eb0 con 0x7f74f80a98f0 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.740+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74f8135610 con 0x7f74f80a98f0 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.740+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f74f4001b00 con 0x7f74f80a98f0 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.740+0000 7f74eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.740+0000 7f74eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.740+0000 7f74eeffd640 10 monclient(hunting): dump: 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:57.741 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:57.742 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:57.742 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:57.742 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:57.742 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:57.742 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:57.742 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:57.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.740+0000 7f74eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.756+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647592284 0 0) 0x7f74f8137230 con 0x7f74f81430c0 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.756+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74dc004eb0 con 0x7f74f81430c0 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.756+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f74f0001b50 con 0x7f74f81430c0 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.756+0000 7f74eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.756+0000 7f74eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.756+0000 7f74eeffd640 10 monclient(hunting): dump: 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:57.757 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:57.758 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:57.758 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:57.758 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:57.758 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:57.758 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:57.758 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:57.758 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:57.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 446619754 0 0) 0x7f74f8135610 con 0x7f74f80a98f0 2024-03-25T18:30:57.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 >> v1:172.21.15.102:6790/0 conn(0x7f74f81430c0 legacy=0x7f74f81e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 >> v1:172.21.15.102:6789/0 conn(0x7f74f813f710 legacy=0x7f74f81dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:57.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:57.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 1 monclient: found mon.b 2024-03-25T18:30:57.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:57.764 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f8135e80 con 0x7f74f80a98f0 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f74eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 10 monclient: _renew_subs 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f8131660 con 0x7f74f80a98f0 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 10 monclient: _renew_subs 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f80b7960 con 0x7f74f80a98f0 2024-03-25T18:30:57.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.763+0000 7f75003f4640 1 librados: init done 2024-03-25T18:30:57.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.764+0000 7f75003f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:57.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.764+0000 7f75003f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:57.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.764+0000 7f75003f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:30:57.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:57.764+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f80627b0 con 0x7f74f80a98f0 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.052+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74f40038a0 con 0x7f74f80a98f0 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.052+0000 7f74eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.052+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f74f4003bb0 con 0x7f74f80a98f0 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.052+0000 7f74eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.052+0000 7f74eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.052+0000 7f74eeffd640 10 monclient: dump: 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:30:58.053 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:58.054 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:30:58.054 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:30:58.054 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:30:58.054 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:30:58.054 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:30:58.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.052+0000 7f74fc98d640 4 set_mon_vals no callback set 2024-03-25T18:30:58.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.053+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f74f4019b80 con 0x7f74f80a98f0 2024-03-25T18:30:58.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.054+0000 7f74fd98f640 10 monclient: get_auth_request con 0x7f74dc06df70 auth_method 0 2024-03-25T18:30:58.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.054+0000 7f74eeffd640 10 client.4459.objecter ms_handle_connect 0x7f74dc06df70 2024-03-25T18:30:58.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.060+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f74f40536e0 con 0x7f74f80a98f0 2024-03-25T18:30:58.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.060+0000 7f74eeffd640 10 client.4459.objecter ms_dispatch 0x7f74f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:30:58.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.060+0000 7f74eeffd640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:30:58.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.060+0000 7f74eeffd640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:30:58.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.060+0000 7f74eeffd640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:30:58.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.061+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74f404f020 con 0x7f74f80a98f0 2024-03-25T18:30:58.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.061+0000 7f74eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:58.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.061+0000 7f74eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:58.298 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.298+0000 7f75003f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:58.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.298+0000 7f75003f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:58.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.298+0000 7f75003f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:30:58.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.298+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74f8136010 con 0x7f74f80a98f0 2024-03-25T18:30:58.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.407+0000 7f74eeffd640 1 -- 172.21.15.102:0/3506435250 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f74f8136010 con 0x7f74f80a98f0 2024-03-25T18:30:58.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.407+0000 7f74eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:58.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.407+0000 7f74eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:58.408 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:30:58.408 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:58.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 10 librados: watch_flush enter 2024-03-25T18:30:58.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 10 librados: watch_flush exit 2024-03-25T18:30:58.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 >> v1:172.21.15.146:6800/3848252714 conn(0x7f74dc06df70 legacy=0x7f74dc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 10 monclient: shutdown 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 >> v1:172.21.15.146:6789/0 conn(0x7f74f80a98f0 legacy=0x7f74f8130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 shutdown_connections 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 >> 172.21.15.102:0/3506435250 conn(0x7f74f80b7ea0 msgr2=0x7f74f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 shutdown_connections 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 1 -- 172.21.15.102:0/3506435250 wait complete. 2024-03-25T18:30:58.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:30:58.409+0000 7f75003f4640 1 librados: shutdown 2024-03-25T18:30:58.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:03.739 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:05:19.975 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:03.746 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:05:19.981 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:03.768 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:05:20.004 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:03.794 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:05:20.007 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:03.800 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:05:20.036 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:03.805 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:05:20.041 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:03.805 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:05:20.019 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:03.843 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:05:20.079 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:03.848 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:05:20.061 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:05.229 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:05.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.346+0000 7f4e331e8640 10 monclient: get_monmap_and_config 2024-03-25T18:31:05.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.346+0000 7f4e331e8640 10 monclient: build_initial_monmap 2024-03-25T18:31:05.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.346+0000 7f4e331e8640 10 monclient: monmap: 2024-03-25T18:31:05.347 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:05.347 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:05.347 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:31:05.347079+0000 2024-03-25T18:31:05.347 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:31:05.347079+0000 2024-03-25T18:31:05.348 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:05.348 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:05.348 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:05.348 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:31:05.348 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:31:05.348 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:05.349 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 1 Processor -- start 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 1 -- start start 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 10 monclient: init 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2c0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 10 monclient(hunting): picked mon.noname-c con 0x7f4e2c13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c142520 con 0x7f4e2c0a98f0 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0b28a0 con 0x7f4e2c13f710 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0b2a30 con 0x7f4e2c1430c0 2024-03-25T18:31:05.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:05.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.347+0000 7f4e331e8640 10 monclient(hunting): authenticate will time out at 7696.635848s 2024-03-25T18:31:05.356 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.356+0000 7f4e31785640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4e2c1430c0 0x7f4e2c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:60612/0 (socket says 172.21.15.102:60612) 2024-03-25T18:31:05.357 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.356+0000 7f4e31785640 1 -- 172.21.15.102:0/180382631 learned_addr learned my addr 172.21.15.102:0/180382631 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:05.361 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.360+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221067878 0 0) 0x7f4e2c0b2a30 con 0x7f4e2c1430c0 2024-03-25T18:31:05.361 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.360+0000 7f4e2b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:05.361 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.360+0000 7f4e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:05.361 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.360+0000 7f4e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:05.361 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.360+0000 7f4e2b7fe640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:31:05.361 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.360+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c003520 con 0x7f4e2c1430c0 2024-03-25T18:31:05.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1942727356 0 0) 0x7f4e1c003520 con 0x7f4e2c1430c0 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 >> v1:172.21.15.146:6789/0 conn(0x7f4e2c13f710 legacy=0x7f4e2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 >> v1:172.21.15.102:6789/0 conn(0x7f4e2c0a98f0 legacy=0x7f4e2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0b44c0 con 0x7f4e2c1430c0 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:05.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4e24001a10 con 0x7f4e2c1430c0 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.369+0000 7f4e2b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.370+0000 7f4e2b7fe640 10 monclient: dump: 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:05.372 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:05.373 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:05.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.370+0000 7f4e331e8640 5 monclient: authenticate success, global_id 4442 2024-03-25T18:31:05.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.370+0000 7f4e331e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:05.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.376+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e24001bf0 con 0x7f4e2c1430c0 2024-03-25T18:31:05.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.376+0000 7f4e2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:05.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.376+0000 7f4e2b7fe640 1 -- 172.21.15.102:0/180382631 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4e24002040 con 0x7f4e2c1430c0 2024-03-25T18:31:05.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.376+0000 7f4e331e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:05.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.376+0000 7f4e331e8640 4 set_mon_vals no callback set 2024-03-25T18:31:05.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.376+0000 7f4e2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:05.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.376+0000 7f4e2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:05.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.376+0000 7f4e2b7fe640 10 monclient: dump: 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:05.378 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.377+0000 7f4e331e8640 10 monclient: shutdown 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.377+0000 7f4e331e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.377+0000 7f4e331e8640 1 -- 172.21.15.102:0/180382631 >> v1:172.21.15.102:6790/0 conn(0x7f4e2c1430c0 legacy=0x7f4e2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.377+0000 7f4e331e8640 1 -- 172.21.15.102:0/180382631 shutdown_connections 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.377+0000 7f4e331e8640 1 -- 172.21.15.102:0/180382631 >> 172.21.15.102:0/180382631 conn(0x7f4e2c0b7ea0 msgr2=0x7f4e2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.377+0000 7f4e331e8640 1 -- 172.21.15.102:0/180382631 shutdown_connections 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.377+0000 7f4e331e8640 1 -- 172.21.15.102:0/180382631 wait complete. 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 10 monclient: build_initial_monmap 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 10 monclient: monmap: 2024-03-25T18:31:05.381 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 1 librados: starting msgr at 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 1 librados: starting objecter 2024-03-25T18:31:05.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 1 Processor -- start 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 1 -- start start 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 1 librados: setting wanted keys 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 1 librados: calling monclient init 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.378+0000 7f4e331e8640 10 monclient: init 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 10 monclient(hunting): picked mon.noname-c con 0x7f4e2c13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2c1430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c1d83a0 con 0x7f4e2c1430c0 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0abe50 con 0x7f4e2c0a98f0 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c138f30 con 0x7f4e2c13f710 2024-03-25T18:31:05.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:05.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.379+0000 7f4e331e8640 10 monclient(hunting): authenticate will time out at 7696.667631s 2024-03-25T18:31:05.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.391+0000 7f4e2bfff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4e2c13f710 0x7f4e2c138820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:60630/0 (socket says 172.21.15.102:60630) 2024-03-25T18:31:05.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.391+0000 7f4e2bfff640 1 -- 172.21.15.102:0/2704836127 learned_addr learned my addr 172.21.15.102:0/2704836127 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:05.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.394+0000 7f4e297fa640 10 client.?.objecter ms_handle_connect 0x7f4e2c13f710 2024-03-25T18:31:05.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.394+0000 7f4e297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:05.395 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.395+0000 7f4e297fa640 10 client.?.objecter ms_handle_connect 0x7f4e2c0a98f0 2024-03-25T18:31:05.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.395+0000 7f4e297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:05.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.401+0000 7f4e297fa640 10 client.?.objecter ms_handle_connect 0x7f4e2c1430c0 2024-03-25T18:31:05.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.401+0000 7f4e297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:05.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.403+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512078814 0 0) 0x7f4e2c138f30 con 0x7f4e2c13f710 2024-03-25T18:31:05.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.403+0000 7f4e297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:05.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.403+0000 7f4e297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:05.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.403+0000 7f4e297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:05.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.403+0000 7f4e297fa640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:31:05.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.403+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e10004eb0 con 0x7f4e2c13f710 2024-03-25T18:31:05.413 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.412+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3938523180 0 0) 0x7f4e10004eb0 con 0x7f4e2c13f710 2024-03-25T18:31:05.413 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.412+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e2c138f30 con 0x7f4e2c13f710 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.412+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4e180019d0 con 0x7f4e2c13f710 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.412+0000 7f4e297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.412+0000 7f4e297fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.412+0000 7f4e297fa640 10 monclient(hunting): dump: 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:05.414 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.412+0000 7f4e297fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.413+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403642830 0 0) 0x7f4e2c1d83a0 con 0x7f4e2c1430c0 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.413+0000 7f4e297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.413+0000 7f4e297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.413+0000 7f4e297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.413+0000 7f4e297fa640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:31:05.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.413+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e10004eb0 con 0x7f4e2c1430c0 2024-03-25T18:31:05.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.419+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954475429 0 0) 0x7f4e2c0abe50 con 0x7f4e2c0a98f0 2024-03-25T18:31:05.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.419+0000 7f4e297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:05.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.419+0000 7f4e297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:05.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.419+0000 7f4e297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:05.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.419+0000 7f4e297fa640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:31:05.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.419+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c1d83a0 con 0x7f4e2c0a98f0 2024-03-25T18:31:05.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 719613635 0 0) 0x7f4e2c138f30 con 0x7f4e2c13f710 2024-03-25T18:31:05.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 >> v1:172.21.15.146:6789/0 conn(0x7f4e2c0a98f0 legacy=0x7f4e2c1d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 >> v1:172.21.15.102:6789/0 conn(0x7f4e2c1430c0 legacy=0x7f4e2c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:05.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 10 monclient: _finish_hunting 0 2024-03-25T18:31:05.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 1 monclient: found mon.c 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c1d7490 con 0x7f4e2c13f710 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 10 monclient: _finish_auth 0 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 10 monclient: _check_auth_tickets 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 10 monclient: _renew_subs 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c1d7770 con 0x7f4e2c13f710 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:05.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 10 monclient: _renew_subs 2024-03-25T18:31:05.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:05.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c138240 con 0x7f4e2c13f710 2024-03-25T18:31:05.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.430+0000 7f4e331e8640 1 librados: init done 2024-03-25T18:31:05.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.431+0000 7f4e331e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:05.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.431+0000 7f4e331e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:05.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.431+0000 7f4e331e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:05.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.431+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c0627b0 con 0x7f4e2c13f710 2024-03-25T18:31:05.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.742+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e18001c50 con 0x7f4e2c13f710 2024-03-25T18:31:05.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.742+0000 7f4e297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:05.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.742+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4e18003a60 con 0x7f4e2c13f710 2024-03-25T18:31:05.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.742+0000 7f4e297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:05.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.742+0000 7f4e297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.742+0000 7f4e297fa640 10 monclient: dump: 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:05.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.742+0000 7f4e2affd640 4 set_mon_vals no callback set 2024-03-25T18:31:05.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.743+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f4e18003c80 con 0x7f4e2c13f710 2024-03-25T18:31:05.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.743+0000 7f4e30f84640 10 monclient: get_auth_request con 0x7f4e1006de10 auth_method 0 2024-03-25T18:31:05.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.744+0000 7f4e297fa640 10 client.4445.objecter ms_handle_connect 0x7f4e1006de10 2024-03-25T18:31:05.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.748+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f4e18054820 con 0x7f4e2c13f710 2024-03-25T18:31:05.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.748+0000 7f4e297fa640 10 client.4445.objecter ms_dispatch 0x7f4e2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:05.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.748+0000 7f4e297fa640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:05.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.748+0000 7f4e297fa640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:05.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.748+0000 7f4e297fa640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-25T18:31:05.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.748+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e18050330 con 0x7f4e2c13f710 2024-03-25T18:31:05.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.748+0000 7f4e297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:05.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.748+0000 7f4e297fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:05.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.997+0000 7f4e331e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:05.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.997+0000 7f4e331e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:05.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.997+0000 7f4e331e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:05.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:05.997+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e2c1d7900 con 0x7f4e2c13f710 2024-03-25T18:31:06.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.033+0000 7f4e297fa640 1 -- 172.21.15.102:0/2704836127 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f4e18052260 con 0x7f4e2c13f710 2024-03-25T18:31:06.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.033+0000 7f4e297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:06.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.033+0000 7f4e297fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:06.035 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:31:06.035 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:06.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 10 librados: watch_flush enter 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 10 librados: watch_flush exit 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 10 client.4445.objecter successfully canceled tick 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 >> v1:172.21.15.146:6800/3848252714 conn(0x7f4e1006de10 legacy=0x7f4e100702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 10 monclient: shutdown 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 >> v1:172.21.15.102:6790/0 conn(0x7f4e2c13f710 legacy=0x7f4e2c138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 shutdown_connections 2024-03-25T18:31:06.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 >> 172.21.15.102:0/2704836127 conn(0x7f4e2c0b7ea0 msgr2=0x7f4e2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:06.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 shutdown_connections 2024-03-25T18:31:06.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 1 -- 172.21.15.102:0/2704836127 wait complete. 2024-03-25T18:31:06.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:06.036+0000 7f4e331e8640 1 librados: shutdown 2024-03-25T18:31:06.048 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:12.855 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:12.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.972+0000 7f1f25ac3640 10 monclient: get_monmap_and_config 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.973+0000 7f1f25ac3640 10 monclient: build_initial_monmap 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.973+0000 7f1f25ac3640 10 monclient: monmap: 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:31:12.973680+0000 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:31:12.973680+0000 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:31:12.974 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:12.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.973+0000 7f1f25ac3640 1 Processor -- start 2024-03-25T18:31:12.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.973+0000 7f1f25ac3640 1 -- start start 2024-03-25T18:31:12.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.973+0000 7f1f25ac3640 10 monclient: init 2024-03-25T18:31:12.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:12.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:12.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 10 monclient(hunting): picked mon.noname-c con 0x7f1f200a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:12.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2013f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:12.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 10 monclient(hunting): picked mon.noname-a con 0x7f1f201430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:12.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f20142520 con 0x7f1f201430c0 2024-03-25T18:31:12.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200b28a0 con 0x7f1f200a98f0 2024-03-25T18:31:12.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200b2a30 con 0x7f1f2013f710 2024-03-25T18:31:12.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:12.976 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.974+0000 7f1f25ac3640 10 monclient(hunting): authenticate will time out at 7704.262523s 2024-03-25T18:31:12.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.981+0000 7f1f1ffff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1f201430c0 0x7f1f2014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38752/0 (socket says 172.21.15.102:38752) 2024-03-25T18:31:12.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.981+0000 7f1f1ffff640 1 -- 172.21.15.102:0/2263120596 learned_addr learned my addr 172.21.15.102:0/2263120596 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:12.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.994+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509357895 0 0) 0x7f1f20142520 con 0x7f1f201430c0 2024-03-25T18:31:12.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.994+0000 7f1f1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:12.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.994+0000 7f1f1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:12.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.994+0000 7f1f1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:12.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.994+0000 7f1f1e7fc640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:31:12.995 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.994+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f08003520 con 0x7f1f201430c0 2024-03-25T18:31:12.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.995+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457336691 0 0) 0x7f1f200b28a0 con 0x7f1f200a98f0 2024-03-25T18:31:12.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.995+0000 7f1f1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:31:12.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.995+0000 7f1f1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:12.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.995+0000 7f1f1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:12.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.995+0000 7f1f1e7fc640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:31:12.996 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:12.995+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f20142520 con 0x7f1f200a98f0 2024-03-25T18:31:13.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3234128945 0 0) 0x7f1f08003520 con 0x7f1f201430c0 2024-03-25T18:31:13.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 >> v1:172.21.15.102:6790/0 conn(0x7f1f2013f710 legacy=0x7f1f201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:31:13.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 >> v1:172.21.15.146:6789/0 conn(0x7f1f200a98f0 legacy=0x7f1f200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:13.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:31:13.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200b44c0 con 0x7f1f201430c0 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1f18001a10 con 0x7f1f201430c0 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f25ac3640 5 monclient: authenticate success, global_id 4467 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f25ac3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f1e7fc640 10 monclient: dump: 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:13.005 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.006 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.006 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:13.006 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:13.006 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:13.006 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:13.006 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:13.006 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:13.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.003+0000 7f1f25ac3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.048+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f18001b70 con 0x7f1f201430c0 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.048+0000 7f1f1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.048+0000 7f1f1e7fc640 1 -- 172.21.15.102:0/2263120596 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1f18001f80 con 0x7f1f201430c0 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.048+0000 7f1f25ac3640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.048+0000 7f1f25ac3640 4 set_mon_vals no callback set 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.048+0000 7f1f1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.048+0000 7f1f1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.048+0000 7f1f1e7fc640 10 monclient: dump: 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.049 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:13.050 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:13.050 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:13.050 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:13.050 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:13.050 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:13.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.049+0000 7f1f25ac3640 10 monclient: shutdown 2024-03-25T18:31:13.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.049+0000 7f1f25ac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:13.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.049+0000 7f1f25ac3640 1 -- 172.21.15.102:0/2263120596 >> v1:172.21.15.102:6789/0 conn(0x7f1f201430c0 legacy=0x7f1f2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:13.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.049+0000 7f1f25ac3640 1 -- 172.21.15.102:0/2263120596 shutdown_connections 2024-03-25T18:31:13.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.049+0000 7f1f25ac3640 1 -- 172.21.15.102:0/2263120596 >> 172.21.15.102:0/2263120596 conn(0x7f1f200b7ea0 msgr2=0x7f1f200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:13.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.049+0000 7f1f25ac3640 1 -- 172.21.15.102:0/2263120596 shutdown_connections 2024-03-25T18:31:13.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.049+0000 7f1f25ac3640 1 -- 172.21.15.102:0/2263120596 wait complete. 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 10 monclient: build_initial_monmap 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 10 monclient: monmap: 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 1 librados: starting msgr at 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 1 librados: starting objecter 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 1 Processor -- start 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 1 -- start start 2024-03-25T18:31:13.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 1 librados: setting wanted keys 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 1 librados: calling monclient init 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.050+0000 7f1f25ac3640 10 monclient: init 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 10 monclient(hunting): picked mon.noname-a con 0x7f1f200a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2013f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 10 monclient(hunting): picked mon.noname-c con 0x7f1f201430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f20139e50 con 0x7f1f200a98f0 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f201db990 con 0x7f1f2013f710 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2013aff0 con 0x7f1f201430c0 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:13.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.051+0000 7f1f25ac3640 10 monclient(hunting): authenticate will time out at 7704.339493s 2024-03-25T18:31:13.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.055+0000 7f1f1f7fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1f200a98f0 0x7f1f2012f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38808/0 (socket says 172.21.15.102:38808) 2024-03-25T18:31:13.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.055+0000 7f1f1f7fe640 1 -- 172.21.15.102:0/1709327052 learned_addr learned my addr 172.21.15.102:0/1709327052 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:13.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.056+0000 7f1f07fff640 10 client.?.objecter ms_handle_connect 0x7f1f200a98f0 2024-03-25T18:31:13.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.056+0000 7f1f07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:13.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.064+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321790263 0 0) 0x7f1f20139e50 con 0x7f1f200a98f0 2024-03-25T18:31:13.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.064+0000 7f1f07fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:13.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.064+0000 7f1f07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:13.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.064+0000 7f1f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:13.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.064+0000 7f1f07fff640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:31:13.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.064+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f0c004eb0 con 0x7f1f200a98f0 2024-03-25T18:31:13.065 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.065+0000 7f1f07fff640 10 client.?.objecter ms_handle_connect 0x7f1f2013f710 2024-03-25T18:31:13.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.065+0000 7f1f07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:13.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.068+0000 7f1f07fff640 10 client.?.objecter ms_handle_connect 0x7f1f201430c0 2024-03-25T18:31:13.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.068+0000 7f1f07fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:13.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.081+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310791792 0 0) 0x7f1f201db990 con 0x7f1f2013f710 2024-03-25T18:31:13.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.081+0000 7f1f07fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:13.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.081+0000 7f1f07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:13.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.081+0000 7f1f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:13.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.081+0000 7f1f07fff640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:31:13.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.081+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f20139e50 con 0x7f1f2013f710 2024-03-25T18:31:13.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.084+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643736934 0 0) 0x7f1f2013aff0 con 0x7f1f201430c0 2024-03-25T18:31:13.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.084+0000 7f1f07fff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:13.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.084+0000 7f1f07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:13.085 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.084+0000 7f1f07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:13.085 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.084+0000 7f1f07fff640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:31:13.085 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.084+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f201db990 con 0x7f1f201430c0 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.096+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 229053699 0 0) 0x7f1f20139e50 con 0x7f1f2013f710 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.096+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f2013aff0 con 0x7f1f2013f710 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.096+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1f14001810 con 0x7f1f2013f710 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.096+0000 7f1f07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.096+0000 7f1f07fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.096+0000 7f1f07fff640 10 monclient(hunting): dump: 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.097 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.098 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:13.098 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:13.098 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:13.098 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:13.098 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:13.098 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:13.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.096+0000 7f1f07fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:13.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.101+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287596258 0 0) 0x7f1f0c004eb0 con 0x7f1f200a98f0 2024-03-25T18:31:13.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.101+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f20139e50 con 0x7f1f200a98f0 2024-03-25T18:31:13.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.101+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1f10001a10 con 0x7f1f200a98f0 2024-03-25T18:31:13.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.101+0000 7f1f07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:13.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.101+0000 7f1f07fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:13.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.101+0000 7f1f07fff640 10 monclient(hunting): dump: 2024-03-25T18:31:13.102 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:13.102 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:13.103 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.103 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.103 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:13.103 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:13.103 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:13.103 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:13.103 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:13.103 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:13.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1630205889 0 0) 0x7f1f2013aff0 con 0x7f1f2013f710 2024-03-25T18:31:13.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 >> v1:172.21.15.102:6790/0 conn(0x7f1f201430c0 legacy=0x7f1f201db280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:13.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 >> v1:172.21.15.102:6789/0 conn(0x7f1f200a98f0 legacy=0x7f1f2012f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 1 monclient: found mon.b 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2013b2d0 con 0x7f1f2013f710 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 10 monclient: _finish_auth 0 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 10 monclient: _renew_subs 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2012fa40 con 0x7f1f2013f710 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 10 monclient: _renew_subs 2024-03-25T18:31:13.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:13.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f200b6d70 con 0x7f1f2013f710 2024-03-25T18:31:13.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.104+0000 7f1f25ac3640 1 librados: init done 2024-03-25T18:31:13.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.105+0000 7f1f25ac3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:13.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.105+0000 7f1f25ac3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:13.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.105+0000 7f1f25ac3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:13.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.105+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2013a000 con 0x7f1f2013f710 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.407+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f14001d00 con 0x7f1f2013f710 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.407+0000 7f1f07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.407+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1f140035f0 con 0x7f1f2013f710 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.407+0000 7f1f07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.407+0000 7f1f07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.407+0000 7f1f07fff640 10 monclient: dump: 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.408 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.407+0000 7f1f1dffb640 4 set_mon_vals no callback set 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.407+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f1f14019600 con 0x7f1f2013f710 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.408+0000 7f1f1f7fe640 10 monclient: get_auth_request con 0x7f1f0c06df70 auth_method 0 2024-03-25T18:31:13.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.408+0000 7f1f07fff640 10 client.4471.objecter ms_handle_connect 0x7f1f0c06df70 2024-03-25T18:31:13.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.414+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f1f14053330 con 0x7f1f2013f710 2024-03-25T18:31:13.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.414+0000 7f1f07fff640 10 client.4471.objecter ms_dispatch 0x7f1f20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:13.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.414+0000 7f1f07fff640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:13.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.414+0000 7f1f07fff640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:13.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.414+0000 7f1f07fff640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-25T18:31:13.415 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.414+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f1404ec70 con 0x7f1f2013f710 2024-03-25T18:31:13.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.414+0000 7f1f07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:13.416 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.414+0000 7f1f07fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:13.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.634+0000 7f1f25ac3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:13.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.634+0000 7f1f25ac3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:13.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.634+0000 7f1f25ac3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:13.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.634+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f200627b0 con 0x7f1f2013f710 2024-03-25T18:31:13.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.671+0000 7f1f07fff640 1 -- 172.21.15.102:0/1709327052 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f1f14050d70 con 0x7f1f2013f710 2024-03-25T18:31:13.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.671+0000 7f1f07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:13.672 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.671+0000 7f1f07fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:13.672 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:31:13.672 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:13.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 10 librados: watch_flush enter 2024-03-25T18:31:13.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 10 librados: watch_flush exit 2024-03-25T18:31:13.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-25T18:31:13.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 10 client.4471.objecter successfully canceled tick 2024-03-25T18:31:13.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 >> v1:172.21.15.146:6800/3848252714 conn(0x7f1f0c06df70 legacy=0x7f1f0c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:13.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 10 monclient: shutdown 2024-03-25T18:31:13.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:13.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 >> v1:172.21.15.146:6789/0 conn(0x7f1f2013f710 legacy=0x7f1f201dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:13.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 shutdown_connections 2024-03-25T18:31:13.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 >> 172.21.15.102:0/1709327052 conn(0x7f1f200b7ea0 msgr2=0x7f1f20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:13.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 shutdown_connections 2024-03-25T18:31:13.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 1 -- 172.21.15.102:0/1709327052 wait complete. 2024-03-25T18:31:13.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:13.674+0000 7f1f25ac3640 1 librados: shutdown 2024-03-25T18:31:13.685 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:20.492 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:20.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.608+0000 7f2178b72640 10 monclient: get_monmap_and_config 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.608+0000 7f2178b72640 10 monclient: build_initial_monmap 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.608+0000 7f2178b72640 10 monclient: monmap: 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:31:20.608910+0000 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:31:20.608910+0000 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:31:20.609 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.609+0000 7f2178b72640 1 Processor -- start 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.609+0000 7f2178b72640 1 -- start start 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.609+0000 7f2178b72640 10 monclient: init 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.609+0000 7f2178b72640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.609+0000 7f2178b72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.609+0000 7f2178b72640 10 monclient(hunting): picked mon.noname-c con 0x7f21740a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.609+0000 7f2178b72640 10 monclient(hunting): picked mon.noname-b con 0x7f217413f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.610+0000 7f2178b72640 10 monclient(hunting): picked mon.noname-a con 0x7f21741430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.610+0000 7f2178b72640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2174142520 con 0x7f21741430c0 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.610+0000 7f2178b72640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21740b28a0 con 0x7f21740a98f0 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.610+0000 7f2178b72640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21740b2a30 con 0x7f217413f710 2024-03-25T18:31:20.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.610+0000 7f2178b72640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:20.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.610+0000 7f2178b72640 10 monclient(hunting): authenticate will time out at 7711.898092s 2024-03-25T18:31:20.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.623+0000 7f217259c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f21740a98f0 0x7f21740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:41072/0 (socket says 172.21.15.102:41072) 2024-03-25T18:31:20.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.623+0000 7f217259c640 1 -- 172.21.15.102:0/1063461971 learned_addr learned my addr 172.21.15.102:0/1063461971 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:20.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.635+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859333958 0 0) 0x7f21740b28a0 con 0x7f21740a98f0 2024-03-25T18:31:20.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.635+0000 7f2171d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:20.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.635+0000 7f2171d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:20.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.635+0000 7f2171d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:20.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.635+0000 7f2171d9b640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:31:20.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.635+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2160003520 con 0x7f21740a98f0 2024-03-25T18:31:20.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.642+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548972917 0 0) 0x7f2174142520 con 0x7f21741430c0 2024-03-25T18:31:20.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.642+0000 7f2171d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:20.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.642+0000 7f2171d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:20.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.642+0000 7f2171d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:20.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.642+0000 7f2171d9b640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:31:20.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.642+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21740b28a0 con 0x7f21741430c0 2024-03-25T18:31:20.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 692630349 0 0) 0x7f2160003520 con 0x7f21740a98f0 2024-03-25T18:31:20.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 >> v1:172.21.15.102:6790/0 conn(0x7f217413f710 legacy=0x7f21741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 >> v1:172.21.15.102:6789/0 conn(0x7f21741430c0 legacy=0x7f217414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:20.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 1 monclient: found mon.noname-c 2024-03-25T18:31:20.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:20.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:31:20.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21740b44c0 con 0x7f21740a98f0 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2164001690 con 0x7f21740a98f0 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2171d9b640 10 monclient: dump: 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:20.649 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:20.650 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:20.650 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:20.650 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:20.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2178b72640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:31:20.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.647+0000 7f2178b72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.657+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21640017f0 con 0x7f21740a98f0 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.657+0000 7f2171d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.657+0000 7f2171d9b640 1 -- 172.21.15.102:0/1063461971 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2164001c00 con 0x7f21740a98f0 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.657+0000 7f2171d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.657+0000 7f2171d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.657+0000 7f2171d9b640 10 monclient: dump: 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:20.658 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.657+0000 7f2178b72640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:20.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.657+0000 7f2178b72640 4 set_mon_vals no callback set 2024-03-25T18:31:20.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.658+0000 7f2178b72640 10 monclient: shutdown 2024-03-25T18:31:20.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.658+0000 7f2178b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:20.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.658+0000 7f2178b72640 1 -- 172.21.15.102:0/1063461971 >> v1:172.21.15.146:6789/0 conn(0x7f21740a98f0 legacy=0x7f21740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.658+0000 7f2178b72640 1 -- 172.21.15.102:0/1063461971 shutdown_connections 2024-03-25T18:31:20.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.658+0000 7f2178b72640 1 -- 172.21.15.102:0/1063461971 >> 172.21.15.102:0/1063461971 conn(0x7f21740b7ea0 msgr2=0x7f21740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:20.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.658+0000 7f2178b72640 1 -- 172.21.15.102:0/1063461971 shutdown_connections 2024-03-25T18:31:20.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.658+0000 7f2178b72640 1 -- 172.21.15.102:0/1063461971 wait complete. 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 10 monclient: build_initial_monmap 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 10 monclient: monmap: 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:20.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 1 librados: starting msgr at 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 1 librados: starting objecter 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 1 Processor -- start 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 1 -- start start 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 1 librados: setting wanted keys 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 1 librados: calling monclient init 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.659+0000 7f2178b72640 10 monclient: init 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 10 monclient(hunting): picked mon.noname-b con 0x7f21740a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 10 monclient(hunting): picked mon.noname-a con 0x7f217413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 10 monclient(hunting): picked mon.noname-c con 0x7f21741430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2174138250 con 0x7f217413f710 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2174137d50 con 0x7f21740a98f0 2024-03-25T18:31:20.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2174137ee0 con 0x7f21741430c0 2024-03-25T18:31:20.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:20.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.660+0000 7f2178b72640 10 monclient(hunting): authenticate will time out at 7711.948507s 2024-03-25T18:31:20.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.671+0000 7f216bfff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f217413f710 0x7f21741d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43690/0 (socket says 172.21.15.102:43690) 2024-03-25T18:31:20.672 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.671+0000 7f216bfff640 1 -- 172.21.15.102:0/991264432 learned_addr learned my addr 172.21.15.102:0/991264432 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:20.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.672+0000 7f216affd640 10 client.?.objecter ms_handle_connect 0x7f217413f710 2024-03-25T18:31:20.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.672+0000 7f216affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:20.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.675+0000 7f216affd640 10 client.?.objecter ms_handle_connect 0x7f21740a98f0 2024-03-25T18:31:20.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.675+0000 7f216affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:20.679 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.679+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898287768 0 0) 0x7f2174138250 con 0x7f217413f710 2024-03-25T18:31:20.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.679+0000 7f216affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:20.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.679+0000 7f216affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:20.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.679+0000 7f216affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:20.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.679+0000 7f216affd640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:31:20.680 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.679+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2154004eb0 con 0x7f217413f710 2024-03-25T18:31:20.685 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.685+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568365024 0 0) 0x7f2174137d50 con 0x7f21740a98f0 2024-03-25T18:31:20.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.685+0000 7f216affd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:20.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.685+0000 7f216affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:20.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.685+0000 7f216affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:20.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.685+0000 7f216affd640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:31:20.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.685+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2174138250 con 0x7f21740a98f0 2024-03-25T18:31:20.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.688+0000 7f216affd640 10 client.?.objecter ms_handle_connect 0x7f21741430c0 2024-03-25T18:31:20.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.688+0000 7f216affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.696+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2713267630 0 0) 0x7f2174138250 con 0x7f21740a98f0 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.696+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2174137d50 con 0x7f21740a98f0 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.696+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2164001b00 con 0x7f21740a98f0 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.696+0000 7f216affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.696+0000 7f216affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.696+0000 7f216affd640 10 monclient(hunting): dump: 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:20.697 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.698 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:20.698 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:20.698 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:20.698 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:20.698 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:20.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.696+0000 7f216affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2281111679 0 0) 0x7f2154004eb0 con 0x7f217413f710 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2174138250 con 0x7f217413f710 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f215c001a30 con 0x7f217413f710 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 10 monclient(hunting): dump: 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:20.707 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:20.708 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:20.708 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:20.708 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:20.708 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:20.708 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:20.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 699318530 0 0) 0x7f2174137d50 con 0x7f21740a98f0 2024-03-25T18:31:20.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 >> v1:172.21.15.102:6790/0 conn(0x7f21741430c0 legacy=0x7f21741d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 >> v1:172.21.15.102:6789/0 conn(0x7f217413f710 legacy=0x7f21741d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:20.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:20.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 1 monclient: found mon.b 2024-03-25T18:31:20.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:20.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21740b7730 con 0x7f21740a98f0 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 10 monclient: _finish_auth 0 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f216affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.706+0000 7f2178b72640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 10 monclient: _renew_subs 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21740b7920 con 0x7f21740a98f0 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 10 monclient: _renew_subs 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21741e33d0 con 0x7f21740a98f0 2024-03-25T18:31:20.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 1 librados: init done 2024-03-25T18:31:20.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.707+0000 7f2178b72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:20.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.708+0000 7f2178b72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:20.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.708+0000 7f2178b72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:20.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:20.708+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21740627b0 con 0x7f21740a98f0 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.022+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2164004a90 con 0x7f21740a98f0 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.022+0000 7f216affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.022+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2164004da0 con 0x7f21740a98f0 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.022+0000 7f216affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.022+0000 7f216affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.022+0000 7f216affd640 10 monclient: dump: 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:21.023 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:21.024 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:21.024 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:21.024 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:21.024 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:21.024 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:21.024 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:21.024 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.022+0000 7f217159a640 4 set_mon_vals no callback set 2024-03-25T18:31:21.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.023+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f216401ad70 con 0x7f21740a98f0 2024-03-25T18:31:21.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.023+0000 7f216bfff640 10 monclient: get_auth_request con 0x7f215406de90 auth_method 0 2024-03-25T18:31:21.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.024+0000 7f216affd640 10 client.4477.objecter ms_handle_connect 0x7f215406de90 2024-03-25T18:31:21.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.028+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f2164054870 con 0x7f21740a98f0 2024-03-25T18:31:21.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.028+0000 7f216affd640 10 client.4477.objecter ms_dispatch 0x7f2174000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:21.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.028+0000 7f216affd640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:21.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.028+0000 7f216affd640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:21.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.028+0000 7f216affd640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T18:31:21.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.028+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2164050380 con 0x7f21740a98f0 2024-03-25T18:31:21.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.028+0000 7f216affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:21.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.028+0000 7f216affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:21.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.288+0000 7f2178b72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:21.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.289+0000 7f2178b72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:21.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.289+0000 7f2178b72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:21.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.289+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f217413df50 con 0x7f21740a98f0 2024-03-25T18:31:21.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.330+0000 7f216affd640 1 -- 172.21.15.102:0/991264432 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f21640522b0 con 0x7f21740a98f0 2024-03-25T18:31:21.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.330+0000 7f216affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:21.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.330+0000 7f216affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:21.331 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:31:21.331 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:21.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 10 librados: watch_flush enter 2024-03-25T18:31:21.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 10 librados: watch_flush exit 2024-03-25T18:31:21.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T18:31:21.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 10 client.4477.objecter successfully canceled tick 2024-03-25T18:31:21.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 >> v1:172.21.15.146:6800/3848252714 conn(0x7f215406de90 legacy=0x7f2154070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:21.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 10 monclient: shutdown 2024-03-25T18:31:21.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:21.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 >> v1:172.21.15.146:6789/0 conn(0x7f21740a98f0 legacy=0x7f2174137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:21.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 shutdown_connections 2024-03-25T18:31:21.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 >> 172.21.15.102:0/991264432 conn(0x7f21740b7ea0 msgr2=0x7f2174074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:21.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 shutdown_connections 2024-03-25T18:31:21.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 1 -- 172.21.15.102:0/991264432 wait complete. 2024-03-25T18:31:21.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:21.332+0000 7f2178b72640 1 librados: shutdown 2024-03-25T18:31:21.344 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:23.814 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:05:40.049 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:23.822 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:05:40.058 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:23.849 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:05:40.085 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:23.872 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:05:40.085 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:23.878 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:05:40.091 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:23.880 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:05:40.116 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:23.886 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:05:40.122 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:23.905 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:05:40.118 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:23.916 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:05:40.152 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:28.151 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:28.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.249+0000 7fee1307d640 10 monclient: get_monmap_and_config 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.249+0000 7fee1307d640 10 monclient: build_initial_monmap 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.249+0000 7fee1307d640 10 monclient: monmap: 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:31:28.250806+0000 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:31:28.250806+0000 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:31:28.251 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:28.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.249+0000 7fee1307d640 1 Processor -- start 2024-03-25T18:31:28.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.249+0000 7fee1307d640 1 -- start start 2024-03-25T18:31:28.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.249+0000 7fee1307d640 10 monclient: init 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c0a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c1430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c142520 con 0x7fee0c1430c0 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0b28a0 con 0x7fee0c13f710 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0b2a30 con 0x7fee0c0a98f0 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:28.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.250+0000 7fee1307d640 10 monclient(hunting): authenticate will time out at 7719.539588s 2024-03-25T18:31:28.260 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.259+0000 7fee10e19640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fee0c0a98f0 0x7fee0c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:57420/0 (socket says 172.21.15.102:57420) 2024-03-25T18:31:28.261 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.259+0000 7fee10e19640 1 -- 172.21.15.102:0/3301789359 learned_addr learned my addr 172.21.15.102:0/3301789359 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:28.277 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.275+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419309564 0 0) 0x7fee0c0b2a30 con 0x7fee0c0a98f0 2024-03-25T18:31:28.277 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.275+0000 7fee0b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:28.277 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.275+0000 7fee0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:28.277 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.275+0000 7fee0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:28.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.275+0000 7fee0b7fe640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:31:28.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.275+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf4003520 con 0x7fee0c0a98f0 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 258298018 0 0) 0x7fedf4003520 con 0x7fee0c0a98f0 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 >> v1:172.21.15.146:6789/0 conn(0x7fee0c13f710 legacy=0x7fee0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 >> v1:172.21.15.102:6789/0 conn(0x7fee0c1430c0 legacy=0x7fee0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0b44c0 con 0x7fee0c0a98f0 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee04001690 con 0x7fee0c0a98f0 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee1307d640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee1307d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:28.306 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee0b7fe640 10 monclient: dump: 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:28.307 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.304+0000 7fee1307d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.315+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee04001860 con 0x7fee0c0a98f0 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.315+0000 7fee0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.315+0000 7fee0b7fe640 1 -- 172.21.15.102:0/3301789359 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee04001cb0 con 0x7fee0c0a98f0 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.315+0000 7fee1307d640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.315+0000 7fee1307d640 4 set_mon_vals no callback set 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.315+0000 7fee0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.315+0000 7fee0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.315+0000 7fee0b7fe640 10 monclient: dump: 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.317 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:28.318 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:28.318 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:28.318 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:28.318 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:28.318 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.316+0000 7fee1307d640 10 monclient: shutdown 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.316+0000 7fee1307d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.316+0000 7fee1307d640 1 -- 172.21.15.102:0/3301789359 >> v1:172.21.15.102:6790/0 conn(0x7fee0c0a98f0 legacy=0x7fee0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.316+0000 7fee1307d640 1 -- 172.21.15.102:0/3301789359 shutdown_connections 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.316+0000 7fee1307d640 1 -- 172.21.15.102:0/3301789359 >> 172.21.15.102:0/3301789359 conn(0x7fee0c0b7ea0 msgr2=0x7fee0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.316+0000 7fee1307d640 1 -- 172.21.15.102:0/3301789359 shutdown_connections 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.316+0000 7fee1307d640 1 -- 172.21.15.102:0/3301789359 wait complete. 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.317+0000 7fee1307d640 10 monclient: build_initial_monmap 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.317+0000 7fee1307d640 10 monclient: monmap: 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:28.319 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:31:28.320 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:31:28.320 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:28.320 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:31:28.320 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:28.320 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:31:28.320 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:31:28.320 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:28.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 1 librados: starting msgr at 2024-03-25T18:31:28.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 1 librados: starting objecter 2024-03-25T18:31:28.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 1 Processor -- start 2024-03-25T18:31:28.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 1 -- start start 2024-03-25T18:31:28.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 1 librados: setting wanted keys 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 1 librados: calling monclient init 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 10 monclient: init 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.318+0000 7fee1307d640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.319+0000 7fee1307d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c1382b0 con 0x7fee0c13f710 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.319+0000 7fee1307d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c138e30 con 0x7fee0c0a98f0 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.319+0000 7fee1307d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c139180 con 0x7fee0c1430c0 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.319+0000 7fee1307d640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:28.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.319+0000 7fee1307d640 10 monclient(hunting): authenticate will time out at 7719.608060s 2024-03-25T18:31:28.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.333+0000 7fee10e19640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fee0c0a98f0 0x7fee0c1d7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:38130/0 (socket says 172.21.15.102:38130) 2024-03-25T18:31:28.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.333+0000 7fee10e19640 1 -- 172.21.15.102:0/200247279 learned_addr learned my addr 172.21.15.102:0/200247279 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:28.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.334+0000 7fee097fa640 10 client.?.objecter ms_handle_connect 0x7fee0c0a98f0 2024-03-25T18:31:28.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.334+0000 7fee097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:28.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.338+0000 7fee097fa640 10 client.?.objecter ms_handle_connect 0x7fee0c1430c0 2024-03-25T18:31:28.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.338+0000 7fee097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:28.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.339+0000 7fee097fa640 10 client.?.objecter ms_handle_connect 0x7fee0c13f710 2024-03-25T18:31:28.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.339+0000 7fee097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:28.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.344+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156172070 0 0) 0x7fee0c138e30 con 0x7fee0c0a98f0 2024-03-25T18:31:28.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.344+0000 7fee097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:28.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.344+0000 7fee097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:28.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.344+0000 7fee097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:28.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.344+0000 7fee097fa640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:31:28.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.344+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf8004eb0 con 0x7fee0c0a98f0 2024-03-25T18:31:28.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.346+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233514016 0 0) 0x7fee0c1382b0 con 0x7fee0c13f710 2024-03-25T18:31:28.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.346+0000 7fee097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:28.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.346+0000 7fee097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:28.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.346+0000 7fee097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:28.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.346+0000 7fee097fa640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:31:28.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.346+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c138e30 con 0x7fee0c13f710 2024-03-25T18:31:28.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.362+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618773037 0 0) 0x7fee0c139180 con 0x7fee0c1430c0 2024-03-25T18:31:28.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.362+0000 7fee097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:28.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.362+0000 7fee097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:28.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.362+0000 7fee097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:28.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.362+0000 7fee097fa640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:31:28.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.362+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c1382b0 con 0x7fee0c1430c0 2024-03-25T18:31:28.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.364+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 306722890 0 0) 0x7fedf8004eb0 con 0x7fee0c0a98f0 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.364+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee0c139180 con 0x7fee0c0a98f0 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.364+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee04001b70 con 0x7fee0c0a98f0 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.364+0000 7fee097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.364+0000 7fee097fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.364+0000 7fee097fa640 10 monclient(hunting): dump: 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:28.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.364+0000 7fee097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.376+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1602478728 0 0) 0x7fee0c1382b0 con 0x7fee0c1430c0 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.376+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedf8004eb0 con 0x7fee0c1430c0 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.376+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fedfc001a10 con 0x7fee0c1430c0 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.376+0000 7fee097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.376+0000 7fee097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.376+0000 7fee097fa640 10 monclient(hunting): dump: 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.378 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.379 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:28.379 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:28.379 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:28.379 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:28.379 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:28.379 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:28.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2276442577 0 0) 0x7fee0c138e30 con 0x7fee0c13f710 2024-03-25T18:31:28.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee0c1382b0 con 0x7fee0c13f710 2024-03-25T18:31:28.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee00001510 con 0x7fee0c13f710 2024-03-25T18:31:28.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:28.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:28.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 10 monclient(hunting): dump: 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 939424422 0 0) 0x7fee0c139180 con 0x7fee0c0a98f0 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 >> v1:172.21.15.102:6790/0 conn(0x7fee0c1430c0 legacy=0x7fee0c1e1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 >> v1:172.21.15.102:6789/0 conn(0x7fee0c13f710 legacy=0x7fee0c138720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 10 monclient: _finish_hunting 0 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 1 monclient: found mon.b 2024-03-25T18:31:28.382 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c137c10 con 0x7fee0c0a98f0 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 10 monclient: _finish_auth 0 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 10 monclient: _check_auth_tickets 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 10 monclient: _renew_subs 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c137e80 con 0x7fee0c0a98f0 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 10 monclient: _renew_subs 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c1e3530 con 0x7fee0c0a98f0 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.378+0000 7fee1307d640 1 librados: init done 2024-03-25T18:31:28.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.379+0000 7fee1307d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:28.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.379+0000 7fee1307d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:28.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.379+0000 7fee1307d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:28.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.379+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c1d7920 con 0x7fee0c0a98f0 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.655+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee04004bf0 con 0x7fee0c0a98f0 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.655+0000 7fee097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.655+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee04004d60 con 0x7fee0c0a98f0 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.655+0000 7fee097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.655+0000 7fee097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.655+0000 7fee097fa640 10 monclient: dump: 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:28.657 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.658 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:28.658 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:28.658 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:28.658 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:28.658 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:28.658 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:28.658 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:28.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.655+0000 7fee0affd640 4 set_mon_vals no callback set 2024-03-25T18:31:28.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.656+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fee0401ad30 con 0x7fee0c0a98f0 2024-03-25T18:31:28.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.657+0000 7fee0bfff640 10 monclient: get_auth_request con 0x7fedf806e0b0 auth_method 0 2024-03-25T18:31:28.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.657+0000 7fee097fa640 10 client.4480.objecter ms_handle_connect 0x7fedf806e0b0 2024-03-25T18:31:28.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.660+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fee04054810 con 0x7fee0c0a98f0 2024-03-25T18:31:28.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.660+0000 7fee097fa640 10 client.4480.objecter ms_dispatch 0x7fee0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:28.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.660+0000 7fee097fa640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:28.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.660+0000 7fee097fa640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:28.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.660+0000 7fee097fa640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T18:31:28.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.660+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee04050320 con 0x7fee0c0a98f0 2024-03-25T18:31:28.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.660+0000 7fee097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:28.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.660+0000 7fee097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:28.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.905+0000 7fee1307d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:28.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.905+0000 7fee1307d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:28.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.905+0000 7fee1307d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:28.907 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.905+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee0c139310 con 0x7fee0c0a98f0 2024-03-25T18:31:28.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.929+0000 7fee097fa640 1 -- 172.21.15.102:0/200247279 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fee04052250 con 0x7fee0c0a98f0 2024-03-25T18:31:28.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.929+0000 7fee097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:28.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.929+0000 7fee097fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:28.931 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:31:28.931 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.931+0000 7fee1307d640 10 librados: watch_flush enter 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.931+0000 7fee1307d640 10 librados: watch_flush exit 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.931+0000 7fee1307d640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.931+0000 7fee1307d640 10 client.4480.objecter successfully canceled tick 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 >> v1:172.21.15.146:6800/3848252714 conn(0x7fedf806e0b0 legacy=0x7fedf8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 10 monclient: shutdown 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 >> v1:172.21.15.146:6789/0 conn(0x7fee0c0a98f0 legacy=0x7fee0c1d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 shutdown_connections 2024-03-25T18:31:28.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 >> 172.21.15.102:0/200247279 conn(0x7fee0c0b7ea0 msgr2=0x7fee0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:28.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 shutdown_connections 2024-03-25T18:31:28.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 1 -- 172.21.15.102:0/200247279 wait complete. 2024-03-25T18:31:28.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:28.932+0000 7fee1307d640 1 librados: shutdown 2024-03-25T18:31:28.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:35.754 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.868+0000 7fa9d495d640 10 monclient: get_monmap_and_config 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.868+0000 7fa9d495d640 10 monclient: build_initial_monmap 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.868+0000 7fa9d495d640 10 monclient: monmap: 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:31:35.869921+0000 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:31:35.869921+0000 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:31:35.870 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 1 Processor -- start 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 1 -- start start 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 10 monclient: init 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d00b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d00b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d00af940 con 0x7fa9d00b1730 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d00b0a70 con 0x7fa9d00b4d60 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d00b0d50 con 0x7fa9d014be90 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:35.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.869+0000 7fa9d495d640 10 monclient(hunting): authenticate will time out at 7727.158828s 2024-03-25T18:31:35.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.871+0000 7fa9ced9d640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa9d014be90 0x7fa9d014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:34426/0 (socket says 172.21.15.102:34426) 2024-03-25T18:31:35.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.871+0000 7fa9ced9d640 1 -- 172.21.15.102:0/3004062588 learned_addr learned my addr 172.21.15.102:0/3004062588 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:35.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.882+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519941285 0 0) 0x7fa9d00b0d50 con 0x7fa9d014be90 2024-03-25T18:31:35.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.882+0000 7fa9cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.882+0000 7fa9cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.882+0000 7fa9cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.882+0000 7fa9cdd9b640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:31:35.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.882+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc003520 con 0x7fa9d014be90 2024-03-25T18:31:35.890 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.889+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325809353 0 0) 0x7fa9d00af940 con 0x7fa9d00b1730 2024-03-25T18:31:35.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.889+0000 7fa9cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.889+0000 7fa9cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.889+0000 7fa9cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.889+0000 7fa9cdd9b640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:31:35.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.889+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d00b0d50 con 0x7fa9d00b1730 2024-03-25T18:31:35.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1034425618 0 0) 0x7fa9bc003520 con 0x7fa9d014be90 2024-03-25T18:31:35.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 >> v1:172.21.15.146:6789/0 conn(0x7fa9d00b4d60 legacy=0x7fa9d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 >> v1:172.21.15.102:6789/0 conn(0x7fa9d00b1730 legacy=0x7fa9d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:31:35.897 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 1 monclient: found mon.noname-b 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d00b1030 con 0x7fa9d014be90 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 10 monclient: _finish_auth 0 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa9c8001a10 con 0x7fa9d014be90 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9cdd9b640 10 monclient: dump: 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.898 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.899 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:35.899 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:35.899 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:35.899 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:35.899 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:35.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9d495d640 5 monclient: authenticate success, global_id 4463 2024-03-25T18:31:35.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.895+0000 7fa9d495d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:35.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.916+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c8001b70 con 0x7fa9d014be90 2024-03-25T18:31:35.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.916+0000 7fa9cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:35.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.916+0000 7fa9cdd9b640 1 -- 172.21.15.102:0/3004062588 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa9c8001f80 con 0x7fa9d014be90 2024-03-25T18:31:35.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.916+0000 7fa9d495d640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:35.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.916+0000 7fa9d495d640 4 set_mon_vals no callback set 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.916+0000 7fa9cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.916+0000 7fa9cdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.916+0000 7fa9cdd9b640 10 monclient: dump: 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:35.919 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:35.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.917+0000 7fa9d495d640 10 monclient: shutdown 2024-03-25T18:31:35.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.917+0000 7fa9d495d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:35.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.917+0000 7fa9d495d640 1 -- 172.21.15.102:0/3004062588 >> v1:172.21.15.102:6790/0 conn(0x7fa9d014be90 legacy=0x7fa9d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.917+0000 7fa9d495d640 1 -- 172.21.15.102:0/3004062588 shutdown_connections 2024-03-25T18:31:35.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.917+0000 7fa9d495d640 1 -- 172.21.15.102:0/3004062588 >> 172.21.15.102:0/3004062588 conn(0x7fa9d00617a0 msgr2=0x7fa9d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:35.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.918+0000 7fa9d495d640 1 -- 172.21.15.102:0/3004062588 shutdown_connections 2024-03-25T18:31:35.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.918+0000 7fa9d495d640 1 -- 172.21.15.102:0/3004062588 wait complete. 2024-03-25T18:31:35.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 10 monclient: build_initial_monmap 2024-03-25T18:31:35.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 10 monclient: monmap: 2024-03-25T18:31:35.921 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:35.921 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 1 librados: starting msgr at 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 1 librados: starting objecter 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 1 Processor -- start 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 1 -- start start 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 1 librados: setting wanted keys 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 1 librados: calling monclient init 2024-03-25T18:31:35.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.919+0000 7fa9d495d640 10 monclient: init 2024-03-25T18:31:35.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:35.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:35.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d00b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:35.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d00b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:35.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:35.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d013b4c0 con 0x7fa9d00b1730 2024-03-25T18:31:35.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0134f90 con 0x7fa9d00b4d60 2024-03-25T18:31:35.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d0141050 con 0x7fa9d014be90 2024-03-25T18:31:35.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:35.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.920+0000 7fa9d495d640 10 monclient(hunting): authenticate will time out at 7727.209922s 2024-03-25T18:31:35.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.928+0000 7fa9ced9d640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fa9d014be90 0x7fa9d0134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:34482/0 (socket says 172.21.15.102:34482) 2024-03-25T18:31:35.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.928+0000 7fa9ced9d640 1 -- 172.21.15.102:0/2932830287 learned_addr learned my addr 172.21.15.102:0/2932830287 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:35.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.930+0000 7fa9c6ffd640 10 client.?.objecter ms_handle_connect 0x7fa9d014be90 2024-03-25T18:31:35.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.930+0000 7fa9c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.935+0000 7fa9c6ffd640 10 client.?.objecter ms_handle_connect 0x7fa9d00b1730 2024-03-25T18:31:35.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.935+0000 7fa9c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.937+0000 7fa9c6ffd640 10 client.?.objecter ms_handle_connect 0x7fa9d00b4d60 2024-03-25T18:31:35.939 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.937+0000 7fa9c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:35.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.943+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825556403 0 0) 0x7fa9d013b4c0 con 0x7fa9d00b1730 2024-03-25T18:31:35.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.943+0000 7fa9c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.943+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.943+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.943+0000 7fa9c6ffd640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:31:35.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.943+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b0004eb0 con 0x7fa9d00b1730 2024-03-25T18:31:35.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.949+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083418602 0 0) 0x7fa9d0134f90 con 0x7fa9d00b4d60 2024-03-25T18:31:35.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.949+0000 7fa9c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.949+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.949+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.949+0000 7fa9c6ffd640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:31:35.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.949+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d013b4c0 con 0x7fa9d00b4d60 2024-03-25T18:31:35.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.954+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901713197 0 0) 0x7fa9d0141050 con 0x7fa9d014be90 2024-03-25T18:31:35.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.954+0000 7fa9c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:35.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.954+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:35.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.954+0000 7fa9c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:35.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.954+0000 7fa9c6ffd640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:31:35.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.954+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d0134f90 con 0x7fa9d014be90 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.965+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3022684773 0 0) 0x7fa9b0004eb0 con 0x7fa9d00b1730 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.965+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d0141050 con 0x7fa9d00b1730 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.965+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa9c0001930 con 0x7fa9d00b1730 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.965+0000 7fa9c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.965+0000 7fa9c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.965+0000 7fa9c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:35.967 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:35.968 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:35.968 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:35.968 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:35.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.965+0000 7fa9c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:35.971 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.969+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2925895095 0 0) 0x7fa9d013b4c0 con 0x7fa9d00b4d60 2024-03-25T18:31:35.971 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.969+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9b0004eb0 con 0x7fa9d00b4d60 2024-03-25T18:31:35.971 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.969+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa9b80018b0 con 0x7fa9d00b4d60 2024-03-25T18:31:35.971 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.969+0000 7fa9c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.971 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.969+0000 7fa9c6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:35.971 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.969+0000 7fa9c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:35.971 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:35.971 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:35.972 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.972 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.972 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.972 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:35.972 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:35.972 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:35.972 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:35.972 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.984+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1393959026 0 0) 0x7fa9d0134f90 con 0x7fa9d014be90 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.984+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d013b4c0 con 0x7fa9d014be90 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.984+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa9c8001e80 con 0x7fa9d014be90 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.984+0000 7fa9c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.984+0000 7fa9c6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.984+0000 7fa9c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:35.986 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.987 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:35.987 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:35.987 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:35.987 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:35.987 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:35.987 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:35.987 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:35.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1048279924 0 0) 0x7fa9d0141050 con 0x7fa9d00b1730 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 >> v1:172.21.15.102:6790/0 conn(0x7fa9d014be90 legacy=0x7fa9d0134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 >> v1:172.21.15.146:6789/0 conn(0x7fa9d00b4d60 legacy=0x7fa9d0134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 1 monclient: found mon.a 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0141330 con 0x7fa9d00b1730 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.987+0000 7fa9d495d640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.988+0000 7fa9d495d640 10 monclient: _renew_subs 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.988+0000 7fa9d495d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:35.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.988+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d0141610 con 0x7fa9d00b1730 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.988+0000 7fa9d495d640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.988+0000 7fa9d495d640 10 monclient: _renew_subs 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.988+0000 7fa9d495d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.988+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d013a990 con 0x7fa9d00b1730 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.988+0000 7fa9d495d640 1 librados: init done 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.989+0000 7fa9d495d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.989+0000 7fa9d495d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.989+0000 7fa9d495d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:35.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:35.989+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d00627b0 con 0x7fa9d00b1730 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.301+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c00035f0 con 0x7fa9d00b1730 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.301+0000 7fa9c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.301+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa9c0003900 con 0x7fa9d00b1730 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.301+0000 7fa9c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.301+0000 7fa9c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.301+0000 7fa9c6ffd640 10 monclient: dump: 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:36.303 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.301+0000 7fa9cd59a640 4 set_mon_vals no callback set 2024-03-25T18:31:36.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.302+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fa9c00198d0 con 0x7fa9d00b1730 2024-03-25T18:31:36.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.302+0000 7fa9c7fff640 10 monclient: get_auth_request con 0x7fa9b006e0b0 auth_method 0 2024-03-25T18:31:36.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.303+0000 7fa9c6ffd640 10 client.4485.objecter ms_handle_connect 0x7fa9b006e0b0 2024-03-25T18:31:36.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.309+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fa9c0053680 con 0x7fa9d00b1730 2024-03-25T18:31:36.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.309+0000 7fa9c6ffd640 10 client.4485.objecter ms_dispatch 0x7fa9d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:36.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.309+0000 7fa9c6ffd640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:36.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.309+0000 7fa9c6ffd640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:36.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.309+0000 7fa9c6ffd640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-25T18:31:36.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.309+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9c004efc0 con 0x7fa9d00b1730 2024-03-25T18:31:36.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.309+0000 7fa9c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:36.311 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.309+0000 7fa9c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:36.531 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.529+0000 7fa9d495d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:36.531 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.529+0000 7fa9d495d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:36.531 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.529+0000 7fa9d495d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:36.531 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.529+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9d0069f40 con 0x7fa9d00b1730 2024-03-25T18:31:36.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.688+0000 7fa9c6ffd640 1 -- 172.21.15.102:0/2932830287 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fa9c00510c0 con 0x7fa9d00b1730 2024-03-25T18:31:36.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.688+0000 7fa9c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:36.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.688+0000 7fa9c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:36.690 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:31:36.690 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:36.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 10 librados: watch_flush enter 2024-03-25T18:31:36.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 10 librados: watch_flush exit 2024-03-25T18:31:36.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-25T18:31:36.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 10 client.4485.objecter successfully canceled tick 2024-03-25T18:31:36.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 >> v1:172.21.15.146:6800/3848252714 conn(0x7fa9b006e0b0 legacy=0x7fa9b0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:36.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 10 monclient: shutdown 2024-03-25T18:31:36.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:36.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 >> v1:172.21.15.102:6789/0 conn(0x7fa9d00b1730 legacy=0x7fa9d0138ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:36.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 shutdown_connections 2024-03-25T18:31:36.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 >> 172.21.15.102:0/2932830287 conn(0x7fa9d00617a0 msgr2=0x7fa9d00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:36.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 shutdown_connections 2024-03-25T18:31:36.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 1 -- 172.21.15.102:0/2932830287 wait complete. 2024-03-25T18:31:36.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:36.690+0000 7fa9d495d640 1 librados: shutdown 2024-03-25T18:31:36.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:43.512 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:43.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.629+0000 7f71bb281640 10 monclient: get_monmap_and_config 2024-03-25T18:31:43.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.629+0000 7f71bb281640 10 monclient: build_initial_monmap 2024-03-25T18:31:43.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.629+0000 7f71bb281640 10 monclient: monmap: 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:31:43.629775+0000 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:31:43.629775+0000 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:43.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.629+0000 7f71bb281640 1 Processor -- start 2024-03-25T18:31:43.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 1 -- start start 2024-03-25T18:31:43.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 10 monclient: init 2024-03-25T18:31:43.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 10 monclient(hunting): picked mon.noname-b con 0x7f71b40a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 10 monclient(hunting): picked mon.noname-c con 0x7f71b413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 10 monclient(hunting): picked mon.noname-a con 0x7f71b41430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b4142520 con 0x7f71b41430c0 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40b28a0 con 0x7f71b413f710 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40b2a30 con 0x7f71b40a98f0 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:43.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.630+0000 7f71bb281640 10 monclient(hunting): authenticate will time out at 7734.918729s 2024-03-25T18:31:43.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.642+0000 7f71b981e640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f71b41430c0 0x7f71b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47252/0 (socket says 172.21.15.102:47252) 2024-03-25T18:31:43.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.642+0000 7f71b981e640 1 -- 172.21.15.102:0/2260842244 learned_addr learned my addr 172.21.15.102:0/2260842244 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:43.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.660+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385979960 0 0) 0x7f71b4142520 con 0x7f71b41430c0 2024-03-25T18:31:43.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.660+0000 7f71abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.660+0000 7f71abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.660+0000 7f71abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.660+0000 7f71abfff640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:31:43.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.660+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a0003520 con 0x7f71b41430c0 2024-03-25T18:31:43.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.670+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112248149 0 0) 0x7f71b40b28a0 con 0x7f71b413f710 2024-03-25T18:31:43.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.670+0000 7f71abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.670+0000 7f71abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.670+0000 7f71abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.670+0000 7f71abfff640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:31:43.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.671+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b4142520 con 0x7f71b413f710 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2122264705 0 0) 0x7f71a0003520 con 0x7f71b41430c0 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 >> v1:172.21.15.102:6790/0 conn(0x7f71b40a98f0 legacy=0x7f71b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 >> v1:172.21.15.146:6789/0 conn(0x7f71b413f710 legacy=0x7f71b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 10 monclient: _finish_hunting 0 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 1 monclient: found mon.noname-a 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b40b44c0 con 0x7f71b41430c0 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 10 monclient: _finish_auth 0 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 10 monclient: _check_auth_tickets 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:43.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f71ac001a10 con 0x7f71b41430c0 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71abfff640 10 monclient: dump: 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71bb281640 5 monclient: authenticate success, global_id 4488 2024-03-25T18:31:43.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.675+0000 7f71bb281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:43.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.693+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ac001b70 con 0x7f71b41430c0 2024-03-25T18:31:43.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.693+0000 7f71abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:43.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.693+0000 7f71abfff640 1 -- 172.21.15.102:0/2260842244 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f71ac001f80 con 0x7f71b41430c0 2024-03-25T18:31:43.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.693+0000 7f71bb281640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:43.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.693+0000 7f71bb281640 4 set_mon_vals no callback set 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.693+0000 7f71abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.693+0000 7f71abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.693+0000 7f71abfff640 10 monclient: dump: 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:43.695 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:43.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.694+0000 7f71bb281640 10 monclient: shutdown 2024-03-25T18:31:43.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.694+0000 7f71bb281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:43.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.694+0000 7f71bb281640 1 -- 172.21.15.102:0/2260842244 >> v1:172.21.15.102:6789/0 conn(0x7f71b41430c0 legacy=0x7f71b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.694+0000 7f71bb281640 1 -- 172.21.15.102:0/2260842244 shutdown_connections 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.694+0000 7f71bb281640 1 -- 172.21.15.102:0/2260842244 >> 172.21.15.102:0/2260842244 conn(0x7f71b40b7ea0 msgr2=0x7f71b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.694+0000 7f71bb281640 1 -- 172.21.15.102:0/2260842244 shutdown_connections 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.694+0000 7f71bb281640 1 -- 172.21.15.102:0/2260842244 wait complete. 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.695+0000 7f71bb281640 10 monclient: build_initial_monmap 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.695+0000 7f71bb281640 10 monclient: monmap: 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:31:43.698 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.695+0000 7f71bb281640 1 librados: starting msgr at 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.695+0000 7f71bb281640 1 librados: starting objecter 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.695+0000 7f71bb281640 1 Processor -- start 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 1 -- start start 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 1 librados: setting wanted keys 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 1 librados: calling monclient init 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 10 monclient: init 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 10 monclient(hunting): picked mon.noname-a con 0x7f71b40a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 10 monclient(hunting): picked mon.noname-b con 0x7f71b413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 10 monclient(hunting): picked mon.noname-c con 0x7f71b41430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b41d7df0 con 0x7f71b40a98f0 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b41d7f80 con 0x7f71b413f710 2024-03-25T18:31:43.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b413b5c0 con 0x7f71b41430c0 2024-03-25T18:31:43.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:43.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.696+0000 7f71bb281640 10 monclient(hunting): authenticate will time out at 7734.984808s 2024-03-25T18:31:43.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.708+0000 7f71b881c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f71b413f710 0x7f71b4133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:35184/0 (socket says 172.21.15.102:35184) 2024-03-25T18:31:43.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.708+0000 7f71b881c640 1 -- 172.21.15.102:0/1928662331 learned_addr learned my addr 172.21.15.102:0/1928662331 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:43.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.709+0000 7f71a9ffb640 10 client.?.objecter ms_handle_connect 0x7f71b413f710 2024-03-25T18:31:43.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.709+0000 7f71a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:43.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.716+0000 7f71a9ffb640 10 client.?.objecter ms_handle_connect 0x7f71b41430c0 2024-03-25T18:31:43.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.716+0000 7f71a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:43.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.717+0000 7f71a9ffb640 10 client.?.objecter ms_handle_connect 0x7f71b40a98f0 2024-03-25T18:31:43.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.717+0000 7f71a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:43.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.718+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256913412 0 0) 0x7f71b413b5c0 con 0x7f71b41430c0 2024-03-25T18:31:43.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.718+0000 7f71a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.718+0000 7f71a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.718+0000 7f71a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.718+0000 7f71a9ffb640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:31:43.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.718+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7198004eb0 con 0x7f71b41430c0 2024-03-25T18:31:43.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.719+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423604818 0 0) 0x7f71b41d7f80 con 0x7f71b413f710 2024-03-25T18:31:43.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.719+0000 7f71a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.719+0000 7f71a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.719+0000 7f71a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.719+0000 7f71a9ffb640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:31:43.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.719+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b413b5c0 con 0x7f71b413f710 2024-03-25T18:31:43.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.725+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176051268 0 0) 0x7f71b41d7df0 con 0x7f71b40a98f0 2024-03-25T18:31:43.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.725+0000 7f71a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:31:43.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.725+0000 7f71a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:43.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.725+0000 7f71a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:43.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.725+0000 7f71a9ffb640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:31:43.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.725+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b41d7f80 con 0x7f71b40a98f0 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.733+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1874020831 0 0) 0x7f7198004eb0 con 0x7f71b41430c0 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.733+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71b41d7df0 con 0x7f71b41430c0 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.733+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f71ac001ec0 con 0x7f71b41430c0 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.733+0000 7f71a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.733+0000 7f71a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.733+0000 7f71a9ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:43.734 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:43.735 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.735 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:43.735 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:43.735 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:43.735 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:43.735 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:43.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.733+0000 7f71a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.735+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4180908297 0 0) 0x7f71b413b5c0 con 0x7f71b413f710 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.735+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7198004eb0 con 0x7f71b413f710 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.735+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f71a40019f0 con 0x7f71b413f710 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.735+0000 7f71a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.735+0000 7f71a9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.735+0000 7f71a9ffb640 10 monclient(hunting): dump: 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:43.736 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:43.737 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:43.737 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:43.737 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:43.737 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:43.737 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:43.750 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 586701541 0 0) 0x7f71b41d7df0 con 0x7f71b41430c0 2024-03-25T18:31:43.750 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 >> v1:172.21.15.146:6789/0 conn(0x7f71b413f710 legacy=0x7f71b4133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.750 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 >> v1:172.21.15.102:6789/0 conn(0x7f71b40a98f0 legacy=0x7f71b41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:43.750 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 1 monclient: found mon.c 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b413b750 con 0x7f71b41430c0 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71bb281640 5 monclient: authenticate success, global_id 4472 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71bb281640 10 monclient: _renew_subs 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71bb281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b4133710 con 0x7f71b41430c0 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.749+0000 7f71bb281640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.750+0000 7f71bb281640 10 monclient: _renew_subs 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.750+0000 7f71bb281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.750+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b41357a0 con 0x7f71b41430c0 2024-03-25T18:31:43.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.750+0000 7f71bb281640 1 librados: init done 2024-03-25T18:31:43.752 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.751+0000 7f71bb281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:43.752 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.751+0000 7f71bb281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:43.752 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.751+0000 7f71bb281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:43.752 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:43.751+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b4138fd0 con 0x7f71b41430c0 2024-03-25T18:31:44.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.094+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ac0020e0 con 0x7f71b41430c0 2024-03-25T18:31:44.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.094+0000 7f71a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:44.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.094+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f71ac004e10 con 0x7f71b41430c0 2024-03-25T18:31:44.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.094+0000 7f71a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:44.095 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.094+0000 7f71a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.094+0000 7f71a9ffb640 10 monclient: dump: 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:44.096 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.094+0000 7f71ab7fe640 4 set_mon_vals no callback set 2024-03-25T18:31:44.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.095+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f71ac01ade0 con 0x7f71b41430c0 2024-03-25T18:31:44.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.096+0000 7f71b901d640 10 monclient: get_auth_request con 0x7f719806df70 auth_method 0 2024-03-25T18:31:44.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.096+0000 7f71a9ffb640 10 client.4472.objecter ms_handle_connect 0x7f719806df70 2024-03-25T18:31:44.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.099+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f71ac0549f0 con 0x7f71b41430c0 2024-03-25T18:31:44.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.099+0000 7f71a9ffb640 10 client.4472.objecter ms_dispatch 0x7f71b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:44.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.099+0000 7f71a9ffb640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:44.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.099+0000 7f71a9ffb640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:44.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.099+0000 7f71a9ffb640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-25T18:31:44.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.099+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71ac050330 con 0x7f71b41430c0 2024-03-25T18:31:44.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.100+0000 7f71a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:44.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.100+0000 7f71a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:44.163 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:00.398 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.168 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:00.404 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.195 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:00.431 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.238 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:00.474 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.244 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:00.480 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.283 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:00.519 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.298+0000 7f71bb281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:44.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.298+0000 7f71bb281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:44.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.298+0000 7f71bb281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:31:44.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.298+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71b4074cc0 con 0x7f71b41430c0 2024-03-25T18:31:44.314 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:00.527 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.329 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:00.542 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.346 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:00.559 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:44.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.349+0000 7f71a9ffb640 1 -- 172.21.15.102:0/1928662331 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f71ac052430 con 0x7f71b41430c0 2024-03-25T18:31:44.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.349+0000 7f71a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:44.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.349+0000 7f71a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:44.350 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:31:44.350 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:44.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 10 librados: watch_flush enter 2024-03-25T18:31:44.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 10 librados: watch_flush exit 2024-03-25T18:31:44.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 10 client.4472.objecter successfully canceled tick 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 >> v1:172.21.15.146:6800/3848252714 conn(0x7f719806df70 legacy=0x7f7198070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 10 monclient: shutdown 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 >> v1:172.21.15.102:6790/0 conn(0x7f71b41430c0 legacy=0x7f71b4135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 shutdown_connections 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 >> 172.21.15.102:0/1928662331 conn(0x7f71b40b7ea0 msgr2=0x7f71b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 shutdown_connections 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 1 -- 172.21.15.102:0/1928662331 wait complete. 2024-03-25T18:31:44.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:44.351+0000 7f71bb281640 1 librados: shutdown 2024-03-25T18:31:44.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:51.169 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.282+0000 7f2cd5d96640 10 monclient: get_monmap_and_config 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.282+0000 7f2cd5d96640 10 monclient: build_initial_monmap 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.282+0000 7f2cd5d96640 10 monclient: monmap: 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:31:51.282988+0000 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:31:51.282988+0000 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:31:51.283 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:51.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.283+0000 7f2cd5d96640 1 Processor -- start 2024-03-25T18:31:51.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.283+0000 7f2cd5d96640 1 -- start start 2024-03-25T18:31:51.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.283+0000 7f2cd5d96640 10 monclient: init 2024-03-25T18:31:51.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:51.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:31:51.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd00b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:51.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd00b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:51.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd014be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:51.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd00af940 con 0x7f2cd00b4d60 2024-03-25T18:31:51.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd00b0a70 con 0x7f2cd014be90 2024-03-25T18:31:51.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd00b0d50 con 0x7f2cd00b1730 2024-03-25T18:31:51.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:51.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.284+0000 7f2cd5d96640 10 monclient(hunting): authenticate will time out at 7742.572265s 2024-03-25T18:31:51.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.295+0000 7f2ccffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2cd014be90 0x7f2cd014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:39930/0 (socket says 172.21.15.102:39930) 2024-03-25T18:31:51.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.295+0000 7f2ccffff640 1 -- 172.21.15.102:0/2276630548 learned_addr learned my addr 172.21.15.102:0/2276630548 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:51.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.311+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500537154 0 0) 0x7f2cd00b0a70 con 0x7f2cd014be90 2024-03-25T18:31:51.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.311+0000 7f2cceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:51.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.311+0000 7f2cceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:51.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.311+0000 7f2cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:51.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.311+0000 7f2cceffd640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:31:51.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.311+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cbc003520 con 0x7f2cd014be90 2024-03-25T18:31:51.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.320+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34018584 0 0) 0x7f2cd00b0d50 con 0x7f2cd00b1730 2024-03-25T18:31:51.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.320+0000 7f2cceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:51.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.320+0000 7f2cceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:51.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.320+0000 7f2cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:51.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.320+0000 7f2cceffd640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:31:51.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.320+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd00b0a70 con 0x7f2cd00b1730 2024-03-25T18:31:51.328 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.328+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297084074 0 0) 0x7f2cd00af940 con 0x7f2cd00b4d60 2024-03-25T18:31:51.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.328+0000 7f2cceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:31:51.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.328+0000 7f2cceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:51.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.328+0000 7f2cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:51.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.328+0000 7f2cceffd640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:31:51.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.328+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd00b0d50 con 0x7f2cd00b4d60 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 693724849 0 0) 0x7f2cd00b0a70 con 0x7f2cd00b1730 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 >> v1:172.21.15.146:6789/0 conn(0x7f2cd014be90 legacy=0x7f2cd014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 >> v1:172.21.15.102:6789/0 conn(0x7f2cd00b4d60 legacy=0x7f2cd013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 1 monclient: found mon.noname-b 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd00b1030 con 0x7f2cd00b1730 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 10 monclient: _finish_auth 0 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:51.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2cb8001670 con 0x7f2cd00b1730 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cceffd640 10 monclient: dump: 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cd5d96640 5 monclient: authenticate success, global_id 4475 2024-03-25T18:31:51.332 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.330+0000 7f2cd5d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb8001850 con 0x7f2cd00b1730 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cceffd640 1 -- 172.21.15.102:0/2276630548 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2cb8001c60 con 0x7f2cd00b1730 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cceffd640 10 monclient: dump: 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:51.343 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:51.344 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:51.344 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:51.344 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cd5d96640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:51.344 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cd5d96640 4 set_mon_vals no callback set 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cd5d96640 10 monclient: shutdown 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cd5d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cd5d96640 1 -- 172.21.15.102:0/2276630548 >> v1:172.21.15.102:6790/0 conn(0x7f2cd00b1730 legacy=0x7f2cd00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cd5d96640 1 -- 172.21.15.102:0/2276630548 shutdown_connections 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.342+0000 7f2cd5d96640 1 -- 172.21.15.102:0/2276630548 >> 172.21.15.102:0/2276630548 conn(0x7f2cd00617a0 msgr2=0x7f2cd0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.343+0000 7f2cd5d96640 1 -- 172.21.15.102:0/2276630548 shutdown_connections 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.343+0000 7f2cd5d96640 1 -- 172.21.15.102:0/2276630548 wait complete. 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.343+0000 7f2cd5d96640 10 monclient: build_initial_monmap 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.343+0000 7f2cd5d96640 10 monclient: monmap: 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:31:51.346 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.344+0000 7f2cd5d96640 1 librados: starting msgr at 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.344+0000 7f2cd5d96640 1 librados: starting objecter 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.344+0000 7f2cd5d96640 1 Processor -- start 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.344+0000 7f2cd5d96640 1 -- start start 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.344+0000 7f2cd5d96640 1 librados: setting wanted keys 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.344+0000 7f2cd5d96640 1 librados: calling monclient init 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.344+0000 7f2cd5d96640 10 monclient: init 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:51.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd00b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd00b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd01381d0 con 0x7f2cd00b1730 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd01397e0 con 0x7f2cd00b4d60 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd0137cd0 con 0x7f2cd014be90 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.345+0000 7f2cd5d96640 10 monclient(hunting): authenticate will time out at 7742.633137s 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.347+0000 7f2ccf7fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2cd00b1730 0x7f2cd0135fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51870/0 (socket says 172.21.15.102:51870) 2024-03-25T18:31:51.348 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.347+0000 7f2ccf7fe640 1 -- 172.21.15.102:0/872984806 learned_addr learned my addr 172.21.15.102:0/872984806 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:51.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.350+0000 7f2cccff9640 10 client.?.objecter ms_handle_connect 0x7f2cd014be90 2024-03-25T18:31:51.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.350+0000 7f2cccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:51.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.352+0000 7f2cccff9640 10 client.?.objecter ms_handle_connect 0x7f2cd00b1730 2024-03-25T18:31:51.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.352+0000 7f2cccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:51.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.358+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580694997 0 0) 0x7f2cd0137cd0 con 0x7f2cd014be90 2024-03-25T18:31:51.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.358+0000 7f2cccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:51.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.358+0000 7f2cccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:51.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.358+0000 7f2cccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:51.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.358+0000 7f2cccff9640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:31:51.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.358+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb0004eb0 con 0x7f2cd014be90 2024-03-25T18:31:51.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.365+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210252435 0 0) 0x7f2cd01381d0 con 0x7f2cd00b1730 2024-03-25T18:31:51.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.365+0000 7f2cccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:51.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.365+0000 7f2cccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:51.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.365+0000 7f2cccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:51.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.365+0000 7f2cccff9640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:31:51.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.365+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd0137cd0 con 0x7f2cd00b1730 2024-03-25T18:31:51.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.366+0000 7f2cccff9640 10 client.?.objecter ms_handle_connect 0x7f2cd00b4d60 2024-03-25T18:31:51.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.366+0000 7f2cccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:51.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.378+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2708635884 0 0) 0x7f2cd0137cd0 con 0x7f2cd00b1730 2024-03-25T18:31:51.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.378+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cd01381d0 con 0x7f2cd00b1730 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.379+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2cc00019f0 con 0x7f2cd00b1730 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.379+0000 7f2cccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.379+0000 7f2cccff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.379+0000 7f2cccff9640 10 monclient(hunting): dump: 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:51.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.379+0000 7f2cccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:51.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.383+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181904619 0 0) 0x7f2cd01397e0 con 0x7f2cd00b4d60 2024-03-25T18:31:51.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.383+0000 7f2cccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:31:51.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.383+0000 7f2cccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:51.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.383+0000 7f2cccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:51.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.383+0000 7f2cccff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:31:51.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.383+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd0137cd0 con 0x7f2cd00b4d60 2024-03-25T18:31:51.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.386+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122848129 0 0) 0x7f2cb0004eb0 con 0x7f2cd014be90 2024-03-25T18:31:51.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.386+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cd01397e0 con 0x7f2cd014be90 2024-03-25T18:31:51.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.386+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2cc8001af0 con 0x7f2cd014be90 2024-03-25T18:31:51.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.386+0000 7f2cccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:51.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.386+0000 7f2cccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:31:51.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.386+0000 7f2cccff9640 10 monclient(hunting): dump: 2024-03-25T18:31:51.387 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:51.388 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.401+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 309112893 0 0) 0x7f2cd0137cd0 con 0x7f2cd00b4d60 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.401+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb0004eb0 con 0x7f2cd00b4d60 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.401+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2cb8001b60 con 0x7f2cd00b4d60 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.401+0000 7f2cccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.401+0000 7f2cccff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.401+0000 7f2cccff9640 10 monclient(hunting): dump: 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:51.402 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:51.403 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2100603335 0 0) 0x7f2cd01381d0 con 0x7f2cd00b1730 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 >> v1:172.21.15.102:6790/0 conn(0x7f2cd014be90 legacy=0x7f2cd01e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 >> v1:172.21.15.146:6789/0 conn(0x7f2cd00b4d60 legacy=0x7f2cd01366e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 10 monclient: _finish_hunting 0 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 1 monclient: found mon.a 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd0137e60 con 0x7f2cd00b1730 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 10 monclient: _finish_auth 0 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 10 monclient: _check_auth_tickets 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 5 monclient: authenticate success, global_id 4497 2024-03-25T18:31:51.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 10 monclient: _renew_subs 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd0138df0 con 0x7f2cd00b1730 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 10 monclient: _renew_subs 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd0145310 con 0x7f2cd00b1730 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.403+0000 7f2cd5d96640 1 librados: init done 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.404+0000 7f2cd5d96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.404+0000 7f2cd5d96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:51.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.404+0000 7f2cd5d96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:51.406 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.404+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd01401f0 con 0x7f2cd00b1730 2024-03-25T18:31:51.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.708+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cc00035f0 con 0x7f2cd00b1730 2024-03-25T18:31:51.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.708+0000 7f2cccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.708+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2cc0003940 con 0x7f2cd00b1730 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.708+0000 7f2cccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.708+0000 7f2cccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.708+0000 7f2cccff9640 10 monclient: dump: 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:51.710 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:51.711 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:51.711 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:51.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.708+0000 7f2cce7fc640 4 set_mon_vals no callback set 2024-03-25T18:31:51.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.709+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f2cc0003b60 con 0x7f2cd00b1730 2024-03-25T18:31:51.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.710+0000 7f2cc7fff640 10 monclient: get_auth_request con 0x7f2cb006e160 auth_method 0 2024-03-25T18:31:51.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.710+0000 7f2cccff9640 10 client.4497.objecter ms_handle_connect 0x7f2cb006e160 2024-03-25T18:31:51.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.719+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f2cc0054770 con 0x7f2cd00b1730 2024-03-25T18:31:51.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.719+0000 7f2cccff9640 10 client.4497.objecter ms_dispatch 0x7f2cd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:51.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.719+0000 7f2cccff9640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:51.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.719+0000 7f2cccff9640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:51.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.719+0000 7f2cccff9640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T18:31:51.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.719+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2cc0050280 con 0x7f2cd00b1730 2024-03-25T18:31:51.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.720+0000 7f2cccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:51.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.720+0000 7f2cccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:51.972 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.971+0000 7f2cd5d96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:51.972 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.971+0000 7f2cd5d96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:51.972 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.971+0000 7f2cd5d96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:31:51.972 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:51.971+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cd00627b0 con 0x7f2cd00b1730 2024-03-25T18:31:52.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.011+0000 7f2cccff9640 1 -- 172.21.15.102:0/872984806 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f2cc00521b0 con 0x7f2cd00b1730 2024-03-25T18:31:52.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.011+0000 7f2cccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:52.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.011+0000 7f2cccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:52.012 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:31:52.013 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:52.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.013+0000 7f2cd5d96640 10 librados: watch_flush enter 2024-03-25T18:31:52.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 10 librados: watch_flush exit 2024-03-25T18:31:52.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T18:31:52.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 10 client.4497.objecter successfully canceled tick 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 >> v1:172.21.15.146:6800/3848252714 conn(0x7f2cb006e160 legacy=0x7f2cb0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 10 monclient: shutdown 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 >> v1:172.21.15.102:6789/0 conn(0x7f2cd00b1730 legacy=0x7f2cd0135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 shutdown_connections 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 >> 172.21.15.102:0/872984806 conn(0x7f2cd00617a0 msgr2=0x7f2cd00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 shutdown_connections 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 1 -- 172.21.15.102:0/872984806 wait complete. 2024-03-25T18:31:52.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:52.014+0000 7f2cd5d96640 1 librados: shutdown 2024-03-25T18:31:52.028 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:31:58.834 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:31:58.953 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.953+0000 7fee1aee4640 10 monclient: get_monmap_and_config 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.953+0000 7fee1aee4640 10 monclient: build_initial_monmap 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.953+0000 7fee1aee4640 10 monclient: monmap: 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:31:58.953814+0000 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:31:58.953814+0000 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:31:58.954 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:58.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 1 Processor -- start 2024-03-25T18:31:58.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 1 -- start start 2024-03-25T18:31:58.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 10 monclient: init 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 10 monclient(hunting): picked mon.noname-a con 0x7fee140b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 10 monclient(hunting): picked mon.noname-c con 0x7fee140b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 10 monclient(hunting): picked mon.noname-b con 0x7fee1414be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140af940 con 0x7fee140b4d60 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140b0a70 con 0x7fee140b1730 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140b0d50 con 0x7fee1414be90 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.954+0000 7fee1aee4640 10 monclient(hunting): authenticate will time out at 7750.242837s 2024-03-25T18:31:58.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.961+0000 7fee18c80640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fee140b4d60 0x7fee1413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40402/0 (socket says 172.21.15.102:40402) 2024-03-25T18:31:58.962 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.961+0000 7fee18c80640 1 -- 172.21.15.102:0/969234384 learned_addr learned my addr 172.21.15.102:0/969234384 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:58.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.978+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047133404 0 0) 0x7fee140af940 con 0x7fee140b4d60 2024-03-25T18:31:58.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.978+0000 7fee137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.978+0000 7fee137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.978+0000 7fee137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.978+0000 7fee137fe640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:31:58.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.978+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee04003520 con 0x7fee140b4d60 2024-03-25T18:31:58.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896073596 0 0) 0x7fee140b0d50 con 0x7fee1414be90 2024-03-25T18:31:58.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:31:58.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee140af940 con 0x7fee1414be90 2024-03-25T18:31:58.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298743606 0 0) 0x7fee140b0a70 con 0x7fee140b1730 2024-03-25T18:31:58.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:31:58.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:58.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:58.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:31:58.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.981+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee140b0d50 con 0x7fee140b1730 2024-03-25T18:31:58.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090789066 0 0) 0x7fee140b0d50 con 0x7fee140b1730 2024-03-25T18:31:58.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 >> v1:172.21.15.102:6790/0 conn(0x7fee1414be90 legacy=0x7fee1414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 >> v1:172.21.15.102:6789/0 conn(0x7fee140b4d60 legacy=0x7fee1413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:58.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:31:58.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 1 monclient: found mon.noname-c 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140b1030 con 0x7fee140b1730 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 10 monclient: _finish_auth 0 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee00001530 con 0x7fee140b1730 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee137fe640 10 monclient: dump: 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:58.999 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:59.000 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:59.000 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:59.000 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:59.000 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:59.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee1aee4640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:31:59.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:58.997+0000 7fee1aee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.013+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee00001690 con 0x7fee140b1730 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.013+0000 7fee137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.013+0000 7fee137fe640 1 -- 172.21.15.102:0/969234384 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee00001aa0 con 0x7fee140b1730 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.013+0000 7fee137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.013+0000 7fee137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.013+0000 7fee137fe640 10 monclient: dump: 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:59.014 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:59.015 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:59.015 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:59.015 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:59.015 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:59.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.013+0000 7fee1aee4640 10 monclient: get_monmap_and_config success 2024-03-25T18:31:59.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.013+0000 7fee1aee4640 4 set_mon_vals no callback set 2024-03-25T18:31:59.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.014+0000 7fee1aee4640 10 monclient: shutdown 2024-03-25T18:31:59.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.014+0000 7fee1aee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:59.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.014+0000 7fee1aee4640 1 -- 172.21.15.102:0/969234384 >> v1:172.21.15.146:6789/0 conn(0x7fee140b1730 legacy=0x7fee140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:59.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.014+0000 7fee1aee4640 1 -- 172.21.15.102:0/969234384 shutdown_connections 2024-03-25T18:31:59.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.014+0000 7fee1aee4640 1 -- 172.21.15.102:0/969234384 >> 172.21.15.102:0/969234384 conn(0x7fee140617a0 msgr2=0x7fee14061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:59.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.014+0000 7fee1aee4640 1 -- 172.21.15.102:0/969234384 shutdown_connections 2024-03-25T18:31:59.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.014+0000 7fee1aee4640 1 -- 172.21.15.102:0/969234384 wait complete. 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.015+0000 7fee1aee4640 10 monclient: build_initial_monmap 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.015+0000 7fee1aee4640 10 monclient: monmap: 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 1 librados: starting msgr at 2024-03-25T18:31:59.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 1 librados: starting objecter 2024-03-25T18:31:59.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 1 Processor -- start 2024-03-25T18:31:59.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 1 -- start start 2024-03-25T18:31:59.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 1 librados: setting wanted keys 2024-03-25T18:31:59.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 1 librados: calling monclient init 2024-03-25T18:31:59.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 10 monclient: init 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 10 monclient(hunting): picked mon.noname-b con 0x7fee140b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.016+0000 7fee1aee4640 10 monclient(hunting): picked mon.noname-c con 0x7fee140b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.017+0000 7fee1aee4640 10 monclient(hunting): picked mon.noname-a con 0x7fee1414be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.017+0000 7fee1aee4640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140a9d70 con 0x7fee1414be90 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.017+0000 7fee1aee4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140aac20 con 0x7fee140b1730 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.017+0000 7fee1aee4640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee140aaf00 con 0x7fee140b4d60 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.017+0000 7fee1aee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:31:59.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.017+0000 7fee1aee4640 10 monclient(hunting): authenticate will time out at 7750.305102s 2024-03-25T18:31:59.024 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.024+0000 7fee18c80640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fee140b1730 0x7fee14142d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:33286/0 (socket says 172.21.15.102:33286) 2024-03-25T18:31:59.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.024+0000 7fee18c80640 1 -- 172.21.15.102:0/3207511461 learned_addr learned my addr 172.21.15.102:0/3207511461 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:31:59.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.030+0000 7fee117fa640 10 client.?.objecter ms_handle_connect 0x7fee140b4d60 2024-03-25T18:31:59.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.030+0000 7fee117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:59.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.030+0000 7fee117fa640 10 client.?.objecter ms_handle_connect 0x7fee140b1730 2024-03-25T18:31:59.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.030+0000 7fee117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:59.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.036+0000 7fee117fa640 10 client.?.objecter ms_handle_connect 0x7fee1414be90 2024-03-25T18:31:59.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.036+0000 7fee117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:31:59.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.041+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406879754 0 0) 0x7fee140aaf00 con 0x7fee140b4d60 2024-03-25T18:31:59.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.041+0000 7fee117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:59.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.041+0000 7fee117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:59.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.041+0000 7fee117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:59.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.041+0000 7fee117fa640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:31:59.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.041+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf8004eb0 con 0x7fee140b4d60 2024-03-25T18:31:59.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.042+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107770726 0 0) 0x7fee140aac20 con 0x7fee140b1730 2024-03-25T18:31:59.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.042+0000 7fee117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:31:59.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.042+0000 7fee117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:31:59.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.042+0000 7fee117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:31:59.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.042+0000 7fee117fa640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:31:59.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.042+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee140aaf00 con 0x7fee140b1730 2024-03-25T18:31:59.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.050+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 129626068 0 0) 0x7fee140aaf00 con 0x7fee140b1730 2024-03-25T18:31:59.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.050+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee140aac20 con 0x7fee140b1730 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.050+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee08001a10 con 0x7fee140b1730 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.050+0000 7fee117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.050+0000 7fee117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.050+0000 7fee117fa640 10 monclient(hunting): dump: 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:59.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.050+0000 7fee117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:31:59.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 661863544 0 0) 0x7fee140aac20 con 0x7fee140b1730 2024-03-25T18:31:59.056 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 >> v1:172.21.15.102:6790/0 conn(0x7fee140b4d60 legacy=0x7fee14140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 >> v1:172.21.15.102:6789/0 conn(0x7fee1414be90 legacy=0x7fee141e27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 10 monclient: _finish_hunting 0 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 1 monclient: found mon.b 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140a9850 con 0x7fee140b1730 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 10 monclient: _finish_auth 0 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 10 monclient: _check_auth_tickets 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 5 monclient: authenticate success, global_id 4504 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 10 monclient: _renew_subs 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee140a9470 con 0x7fee140b1730 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:31:59.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 10 monclient: _renew_subs 2024-03-25T18:31:59.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:59.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee14140c70 con 0x7fee140b1730 2024-03-25T18:31:59.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.055+0000 7fee1aee4640 1 librados: init done 2024-03-25T18:31:59.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.056+0000 7fee1aee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:59.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.056+0000 7fee1aee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:59.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.056+0000 7fee1aee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:59.058 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.056+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee140627b0 con 0x7fee140b1730 2024-03-25T18:31:59.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.328+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee080047e0 con 0x7fee140b1730 2024-03-25T18:31:59.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.328+0000 7fee117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:31:59.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.328+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fee08004af0 con 0x7fee140b1730 2024-03-25T18:31:59.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.328+0000 7fee117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:31:59.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.328+0000 7fee117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:31:59.329 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.328+0000 7fee117fa640 10 monclient: dump: 2024-03-25T18:31:59.329 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:31:59.329 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:31:59.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.328+0000 7fee12ffd640 4 set_mon_vals no callback set 2024-03-25T18:31:59.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.329+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fee0801ab40 con 0x7fee140b1730 2024-03-25T18:31:59.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.330+0000 7fee13fff640 10 monclient: get_auth_request con 0x7fedf806dd90 auth_method 0 2024-03-25T18:31:59.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.330+0000 7fee117fa640 10 client.4504.objecter ms_handle_connect 0x7fedf806dd90 2024-03-25T18:31:59.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.334+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fee08054760 con 0x7fee140b1730 2024-03-25T18:31:59.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.334+0000 7fee117fa640 10 client.4504.objecter ms_dispatch 0x7fee14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:31:59.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.334+0000 7fee117fa640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:31:59.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.334+0000 7fee117fa640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:31:59.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.334+0000 7fee117fa640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T18:31:59.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.334+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee080500a0 con 0x7fee140b1730 2024-03-25T18:31:59.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.334+0000 7fee117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:31:59.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.334+0000 7fee117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:31:59.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.568+0000 7fee1aee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:59.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.568+0000 7fee1aee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:59.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.568+0000 7fee1aee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:31:59.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.568+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee140a99e0 con 0x7fee140b1730 2024-03-25T18:31:59.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.599+0000 7fee117fa640 1 -- 172.21.15.102:0/3207511461 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fee080521a0 con 0x7fee140b1730 2024-03-25T18:31:59.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.599+0000 7fee117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:31:59.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.599+0000 7fee117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:31:59.601 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:31:59.601 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:31:59.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 10 librados: watch_flush enter 2024-03-25T18:31:59.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 10 librados: watch_flush exit 2024-03-25T18:31:59.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T18:31:59.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 10 client.4504.objecter successfully canceled tick 2024-03-25T18:31:59.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 >> v1:172.21.15.146:6800/3848252714 conn(0x7fedf806dd90 legacy=0x7fedf8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:59.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 10 monclient: shutdown 2024-03-25T18:31:59.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:31:59.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 >> v1:172.21.15.146:6789/0 conn(0x7fee140b1730 legacy=0x7fee14142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:31:59.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 shutdown_connections 2024-03-25T18:31:59.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 >> 172.21.15.102:0/3207511461 conn(0x7fee140617a0 msgr2=0x7fee1413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:31:59.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 shutdown_connections 2024-03-25T18:31:59.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 1 -- 172.21.15.102:0/3207511461 wait complete. 2024-03-25T18:31:59.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:31:59.602+0000 7fee1aee4640 1 librados: shutdown 2024-03-25T18:31:59.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:03.885 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:20.120 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:03.938 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:20.174 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:03.983 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:20.219 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:03.987 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:20.200 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:03.994 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:20.230 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:03.997 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:20.233 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:03.998 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:20.211 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:04.028 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:20.264 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:04.035 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:20.248 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:06.423 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.536+0000 7f61ddad6640 10 monclient: get_monmap_and_config 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.536+0000 7f61ddad6640 10 monclient: build_initial_monmap 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.536+0000 7f61ddad6640 10 monclient: monmap: 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:32:06.537039+0000 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:32:06.537039+0000 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:32:06.537 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:32:06.538 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 1 Processor -- start 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 1 -- start start 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 10 monclient: init 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 10 monclient(hunting): picked mon.noname-c con 0x7f61d80a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 10 monclient(hunting): picked mon.noname-a con 0x7f61d813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 10 monclient(hunting): picked mon.noname-b con 0x7f61d81430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d8142520 con 0x7f61d813f710 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d80b28a0 con 0x7f61d80a98f0 2024-03-25T18:32:06.539 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d80b2a30 con 0x7f61d81430c0 2024-03-25T18:32:06.540 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:06.540 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.537+0000 7f61ddad6640 10 monclient(hunting): authenticate will time out at 7757.825908s 2024-03-25T18:32:06.544 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.544+0000 7f61d77fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f61d80a98f0 0x7f61d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:48154/0 (socket says 172.21.15.102:48154) 2024-03-25T18:32:06.544 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.544+0000 7f61d77fe640 1 -- 172.21.15.102:0/2888704051 learned_addr learned my addr 172.21.15.102:0/2888704051 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:06.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.558+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630885611 0 0) 0x7f61d80b28a0 con 0x7f61d80a98f0 2024-03-25T18:32:06.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.559+0000 7f61d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:32:06.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.559+0000 7f61d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:06.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.559+0000 7f61d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:06.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.559+0000 7f61d67fc640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:32:06.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.559+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0003520 con 0x7f61d80a98f0 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 396172736 0 0) 0x7f61c0003520 con 0x7f61d80a98f0 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 >> v1:172.21.15.102:6790/0 conn(0x7f61d81430c0 legacy=0x7f61d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 >> v1:172.21.15.102:6789/0 conn(0x7f61d813f710 legacy=0x7f61d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 1 monclient: found mon.noname-c 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d80b44c0 con 0x7f61d80a98f0 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 10 monclient: _finish_auth 0 2024-03-25T18:32:06.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f61c8001690 con 0x7f61d80a98f0 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61d67fc640 10 monclient: dump: 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:06.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61ddad6640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:32:06.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.583+0000 7f61ddad6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61c80017f0 con 0x7f61d80a98f0 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61d67fc640 1 -- 172.21.15.102:0/2888704051 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f61c8001c40 con 0x7f61d80a98f0 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61d67fc640 10 monclient: dump: 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:06.592 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:06.593 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:06.593 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:06.593 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:06.593 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:06.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61ddad6640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:06.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61ddad6640 4 set_mon_vals no callback set 2024-03-25T18:32:06.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61ddad6640 10 monclient: shutdown 2024-03-25T18:32:06.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61ddad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:06.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.591+0000 7f61ddad6640 1 -- 172.21.15.102:0/2888704051 >> v1:172.21.15.146:6789/0 conn(0x7f61d80a98f0 legacy=0x7f61d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.592+0000 7f61ddad6640 1 -- 172.21.15.102:0/2888704051 shutdown_connections 2024-03-25T18:32:06.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.592+0000 7f61ddad6640 1 -- 172.21.15.102:0/2888704051 >> 172.21.15.102:0/2888704051 conn(0x7f61d80b7ea0 msgr2=0x7f61d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.592+0000 7f61ddad6640 1 -- 172.21.15.102:0/2888704051 shutdown_connections 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.592+0000 7f61ddad6640 1 -- 172.21.15.102:0/2888704051 wait complete. 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.592+0000 7f61ddad6640 10 monclient: build_initial_monmap 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.592+0000 7f61ddad6640 10 monclient: monmap: 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:06.596 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.593+0000 7f61ddad6640 1 librados: starting msgr at 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.593+0000 7f61ddad6640 1 librados: starting objecter 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.593+0000 7f61ddad6640 1 Processor -- start 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.593+0000 7f61ddad6640 1 -- start start 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.593+0000 7f61ddad6640 1 librados: setting wanted keys 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.593+0000 7f61ddad6640 1 librados: calling monclient init 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.593+0000 7f61ddad6640 10 monclient: init 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 10 monclient(hunting): picked mon.noname-b con 0x7f61d80a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 10 monclient(hunting): picked mon.noname-a con 0x7f61d813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 10 monclient(hunting): picked mon.noname-c con 0x7f61d81430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d81db7d0 con 0x7f61d813f710 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d80b1420 con 0x7f61d80a98f0 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d80b15b0 con 0x7f61d81430c0 2024-03-25T18:32:06.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:06.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.594+0000 7f61ddad6640 10 monclient(hunting): authenticate will time out at 7757.882502s 2024-03-25T18:32:06.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.605+0000 7f61d77fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f61d80a98f0 0x7f61d81daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:48198/0 (socket says 172.21.15.102:48198) 2024-03-25T18:32:06.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.605+0000 7f61d77fe640 1 -- 172.21.15.102:0/1820793117 learned_addr learned my addr 172.21.15.102:0/1820793117 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:06.607 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.606+0000 7f61bffff640 10 client.?.objecter ms_handle_connect 0x7f61d80a98f0 2024-03-25T18:32:06.607 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.606+0000 7f61bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:06.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.613+0000 7f61bffff640 10 client.?.objecter ms_handle_connect 0x7f61d813f710 2024-03-25T18:32:06.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.613+0000 7f61bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:06.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.614+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169204856 0 0) 0x7f61d80b1420 con 0x7f61d80a98f0 2024-03-25T18:32:06.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.614+0000 7f61bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:06.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.614+0000 7f61bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:06.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.614+0000 7f61bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:06.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.614+0000 7f61bffff640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:32:06.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.614+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c4004eb0 con 0x7f61d80a98f0 2024-03-25T18:32:06.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.625+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320088415 0 0) 0x7f61d81db7d0 con 0x7f61d813f710 2024-03-25T18:32:06.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.625+0000 7f61bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:06.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.625+0000 7f61bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:06.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.625+0000 7f61bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:06.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.625+0000 7f61bffff640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:32:06.627 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.625+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61d80b1420 con 0x7f61d813f710 2024-03-25T18:32:06.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 10 client.?.objecter ms_handle_connect 0x7f61d81430c0 2024-03-25T18:32:06.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:06.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3902948122 0 0) 0x7f61c4004eb0 con 0x7f61d80a98f0 2024-03-25T18:32:06.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61d81db7d0 con 0x7f61d80a98f0 2024-03-25T18:32:06.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f61c8001b00 con 0x7f61d80a98f0 2024-03-25T18:32:06.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:06.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:06.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 10 monclient(hunting): dump: 2024-03-25T18:32:06.630 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:06.630 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:06.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.628+0000 7f61bffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:06.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.645+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938229413 0 0) 0x7f61d80b15b0 con 0x7f61d81430c0 2024-03-25T18:32:06.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.645+0000 7f61bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:06.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.645+0000 7f61bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:06.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.645+0000 7f61bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:06.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.645+0000 7f61bffff640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:32:06.646 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.645+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c4004eb0 con 0x7f61d81430c0 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.645+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 672398476 0 0) 0x7f61d81db7d0 con 0x7f61d80a98f0 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 >> v1:172.21.15.102:6790/0 conn(0x7f61d81430c0 legacy=0x7f61d81d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 >> v1:172.21.15.102:6789/0 conn(0x7f61d813f710 legacy=0x7f61d80b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 1 monclient: found mon.b 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d80b1740 con 0x7f61d80a98f0 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 10 monclient: _finish_auth 0 2024-03-25T18:32:06.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 10 monclient: _renew_subs 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d81db1a0 con 0x7f61d80a98f0 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 10 monclient: _renew_subs 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d80b08f0 con 0x7f61d80a98f0 2024-03-25T18:32:06.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.646+0000 7f61ddad6640 1 librados: init done 2024-03-25T18:32:06.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.647+0000 7f61ddad6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:06.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.647+0000 7f61ddad6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:06.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.647+0000 7f61ddad6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:06.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.647+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d80627b0 con 0x7f61d80a98f0 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.924+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61c8001d20 con 0x7f61d80a98f0 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.924+0000 7f61bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.924+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f61c8004d90 con 0x7f61d80a98f0 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.924+0000 7f61bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.924+0000 7f61bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.924+0000 7f61bffff640 10 monclient: dump: 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:06.925 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:06.926 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:06.926 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:06.926 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:06.926 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:06.926 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:06.926 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:06.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.924+0000 7f61d5ffb640 4 set_mon_vals no callback set 2024-03-25T18:32:06.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.925+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f61c801ad60 con 0x7f61d80a98f0 2024-03-25T18:32:06.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.925+0000 7f61d6ffd640 10 monclient: get_auth_request con 0x7f61c406ddb0 auth_method 0 2024-03-25T18:32:06.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.925+0000 7f61bffff640 10 client.4510.objecter ms_handle_connect 0x7f61c406ddb0 2024-03-25T18:32:06.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.929+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f61c8054910 con 0x7f61d80a98f0 2024-03-25T18:32:06.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.929+0000 7f61bffff640 10 client.4510.objecter ms_dispatch 0x7f61d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:06.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.929+0000 7f61bffff640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:06.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.929+0000 7f61bffff640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:06.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.929+0000 7f61bffff640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:32:06.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.930+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61c8050250 con 0x7f61d80a98f0 2024-03-25T18:32:06.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.930+0000 7f61bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:06.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:06.930+0000 7f61bffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:07.175 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.174+0000 7f61ddad6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:07.175 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.174+0000 7f61ddad6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:07.175 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.174+0000 7f61ddad6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:07.175 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.174+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61d80abb10 con 0x7f61d80a98f0 2024-03-25T18:32:07.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.265+0000 7f61bffff640 1 -- 172.21.15.102:0/1820793117 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f61c8052350 con 0x7f61d80a98f0 2024-03-25T18:32:07.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.265+0000 7f61bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:07.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.265+0000 7f61bffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:07.266 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:32:07.266 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:07.268 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 10 librados: watch_flush enter 2024-03-25T18:32:07.268 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 10 librados: watch_flush exit 2024-03-25T18:32:07.268 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 >> v1:172.21.15.146:6800/3848252714 conn(0x7f61c406ddb0 legacy=0x7f61c4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 10 monclient: shutdown 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 >> v1:172.21.15.146:6789/0 conn(0x7f61d80a98f0 legacy=0x7f61d81daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 shutdown_connections 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 >> 172.21.15.102:0/1820793117 conn(0x7f61d80b7ea0 msgr2=0x7f61d813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 shutdown_connections 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 1 -- 172.21.15.102:0/1820793117 wait complete. 2024-03-25T18:32:07.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:07.267+0000 7f61ddad6640 1 librados: shutdown 2024-03-25T18:32:07.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:14.086 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:14.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.196+0000 7f6d727fb640 10 monclient: get_monmap_and_config 2024-03-25T18:32:14.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.196+0000 7f6d727fb640 10 monclient: build_initial_monmap 2024-03-25T18:32:14.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.196+0000 7f6d727fb640 10 monclient: monmap: 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:32:14.198423+0000 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:32:14.198423+0000 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:32:14.199 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.197+0000 7f6d727fb640 1 Processor -- start 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.197+0000 7f6d727fb640 1 -- start start 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.197+0000 7f6d727fb640 10 monclient: init 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 10 monclient(hunting): picked mon.noname-b con 0x7f6d6c0b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 10 monclient(hunting): picked mon.noname-c con 0x7f6d6c0b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 10 monclient(hunting): picked mon.noname-a con 0x7f6d6c14be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c0af940 con 0x7f6d6c14be90 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c0b0a70 con 0x7f6d6c0b1730 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c0b0d50 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.198+0000 7f6d727fb640 10 monclient(hunting): authenticate will time out at 7765.487507s 2024-03-25T18:32:14.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.199+0000 7f6d6b7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6d6c0b1730 0x7f6d6c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:58296/0 (socket says 172.21.15.102:58296) 2024-03-25T18:32:14.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.199+0000 7f6d6b7fe640 1 -- 172.21.15.102:0/3655855951 learned_addr learned my addr 172.21.15.102:0/3655855951 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:14.242 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.241+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219275793 0 0) 0x7f6d6c0af940 con 0x7f6d6c14be90 2024-03-25T18:32:14.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.241+0000 7f6d6affd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:14.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.241+0000 7f6d6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:14.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.241+0000 7f6d6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:14.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.241+0000 7f6d6affd640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:32:14.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.241+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d54003520 con 0x7f6d6c14be90 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 782675680 0 0) 0x7f6d54003520 con 0x7f6d6c14be90 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 >> v1:172.21.15.102:6790/0 conn(0x7f6d6c0b4d60 legacy=0x7f6d6c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 >> v1:172.21.15.146:6789/0 conn(0x7f6d6c0b1730 legacy=0x7f6d6c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 1 monclient: found mon.noname-a 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c0b1030 con 0x7f6d6c14be90 2024-03-25T18:32:14.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 10 monclient: _finish_auth 0 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d64001a10 con 0x7f6d6c14be90 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d727fb640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d727fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d6affd640 10 monclient: dump: 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:14.253 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:14.254 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:14.254 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:14.254 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:14.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.250+0000 7f6d727fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.273+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d64001be0 con 0x7f6d6c14be90 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.273+0000 7f6d6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.273+0000 7f6d6affd640 1 -- 172.21.15.102:0/3655855951 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d64002030 con 0x7f6d6c14be90 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.273+0000 7f6d727fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.273+0000 7f6d727fb640 4 set_mon_vals no callback set 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.273+0000 7f6d6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.273+0000 7f6d6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.273+0000 7f6d6affd640 10 monclient: dump: 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:14.275 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:14.276 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.276 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.276 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:14.276 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:14.276 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:14.276 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:14.276 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:14.276 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.274+0000 7f6d727fb640 10 monclient: shutdown 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.274+0000 7f6d727fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.274+0000 7f6d727fb640 1 -- 172.21.15.102:0/3655855951 >> v1:172.21.15.102:6789/0 conn(0x7f6d6c14be90 legacy=0x7f6d6c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.274+0000 7f6d727fb640 1 -- 172.21.15.102:0/3655855951 shutdown_connections 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.274+0000 7f6d727fb640 1 -- 172.21.15.102:0/3655855951 >> 172.21.15.102:0/3655855951 conn(0x7f6d6c0617a0 msgr2=0x7f6d6c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.274+0000 7f6d727fb640 1 -- 172.21.15.102:0/3655855951 shutdown_connections 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.274+0000 7f6d727fb640 1 -- 172.21.15.102:0/3655855951 wait complete. 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.275+0000 7f6d727fb640 10 monclient: build_initial_monmap 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.275+0000 7f6d727fb640 10 monclient: monmap: 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:14.278 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.275+0000 7f6d727fb640 1 librados: starting msgr at 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.275+0000 7f6d727fb640 1 librados: starting objecter 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.275+0000 7f6d727fb640 1 Processor -- start 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.276+0000 7f6d727fb640 1 -- start start 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.276+0000 7f6d727fb640 1 librados: setting wanted keys 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.276+0000 7f6d727fb640 1 librados: calling monclient init 2024-03-25T18:32:14.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.276+0000 7f6d727fb640 10 monclient: init 2024-03-25T18:32:14.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.276+0000 7f6d727fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:14.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.276+0000 7f6d727fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:14.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.276+0000 7f6d727fb640 10 monclient(hunting): picked mon.noname-a con 0x7f6d6c0b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:14.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.276+0000 7f6d727fb640 10 monclient(hunting): picked mon.noname-b con 0x7f6d6c0b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:14.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.277+0000 7f6d727fb640 10 monclient(hunting): picked mon.noname-c con 0x7f6d6c14be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:14.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.277+0000 7f6d727fb640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c13b740 con 0x7f6d6c0b1730 2024-03-25T18:32:14.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.277+0000 7f6d727fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c1d7980 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.277+0000 7f6d727fb640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c139350 con 0x7f6d6c14be90 2024-03-25T18:32:14.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.277+0000 7f6d727fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:14.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.277+0000 7f6d727fb640 10 monclient(hunting): authenticate will time out at 7765.566011s 2024-03-25T18:32:14.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.290+0000 7f6d6bfff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6d6c0b1730 0x7f6d6c1d7ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38958/0 (socket says 172.21.15.102:38958) 2024-03-25T18:32:14.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.290+0000 7f6d6bfff640 1 -- 172.21.15.102:0/949887987 learned_addr learned my addr 172.21.15.102:0/949887987 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:14.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.294+0000 7f6d68ff9640 10 client.?.objecter ms_handle_connect 0x7f6d6c0b1730 2024-03-25T18:32:14.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.294+0000 7f6d68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:14.302 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.300+0000 7f6d68ff9640 10 client.?.objecter ms_handle_connect 0x7f6d6c0b4d60 2024-03-25T18:32:14.302 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.300+0000 7f6d68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:14.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.310+0000 7f6d68ff9640 10 client.?.objecter ms_handle_connect 0x7f6d6c14be90 2024-03-25T18:32:14.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.310+0000 7f6d68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:14.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.310+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968968952 0 0) 0x7f6d6c13b740 con 0x7f6d6c0b1730 2024-03-25T18:32:14.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.310+0000 7f6d68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:14.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.310+0000 7f6d68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:14.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.310+0000 7f6d68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:14.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.310+0000 7f6d68ff9640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:32:14.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.310+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d58004eb0 con 0x7f6d6c0b1730 2024-03-25T18:32:14.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.321+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102619993 0 0) 0x7f6d6c1d7980 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.322+0000 7f6d68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:14.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.322+0000 7f6d68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:14.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.322+0000 7f6d68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:14.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.322+0000 7f6d68ff9640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:32:14.324 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.322+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d6c13b740 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2450507563 0 0) 0x7f6d6c13b740 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d6c1d7980 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d600018a0 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:14.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:14.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:14.333 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:14.333 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632867966 0 0) 0x7f6d6c139350 con 0x7f6d6c14be90 2024-03-25T18:32:14.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:32:14.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:14.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:14.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:32:14.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.331+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d6c13b740 con 0x7f6d6c14be90 2024-03-25T18:32:14.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.336+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 932394375 0 0) 0x7f6d58004eb0 con 0x7f6d6c0b1730 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.336+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d6c139350 con 0x7f6d6c0b1730 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.336+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d5c001960 con 0x7f6d6c0b1730 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.336+0000 7f6d68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.336+0000 7f6d68ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.336+0000 7f6d68ff9640 10 monclient(hunting): dump: 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:14.338 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:14.339 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:14.349 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3157460614 0 0) 0x7f6d6c1d7980 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.349 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 >> v1:172.21.15.102:6790/0 conn(0x7f6d6c14be90 legacy=0x7f6d6c132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.349 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 >> v1:172.21.15.102:6789/0 conn(0x7f6d6c0b1730 legacy=0x7f6d6c1d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.349 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:32:14.349 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 1 monclient: found mon.b 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c143900 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 10 monclient: _finish_auth 0 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 10 monclient: _renew_subs 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6c1d85b0 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 10 monclient: _renew_subs 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6c13a700 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.350 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.347+0000 7f6d727fb640 1 librados: init done 2024-03-25T18:32:14.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.348+0000 7f6d727fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:14.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.348+0000 7f6d727fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:14.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.348+0000 7f6d727fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:14.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.348+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6c1d7b10 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.650+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d60001cb0 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.650+0000 7f6d68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.650+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d60004980 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.651+0000 7f6d68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.651+0000 7f6d68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.651+0000 7f6d68ff9640 10 monclient: dump: 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:14.653 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.650+0000 7f6d6a7fc640 4 set_mon_vals no callback set 2024-03-25T18:32:14.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.651+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f6d60004b60 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.652+0000 7f6d6bfff640 10 monclient: get_auth_request con 0x7f6d5806dfb0 auth_method 0 2024-03-25T18:32:14.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.652+0000 7f6d68ff9640 10 client.4516.objecter ms_handle_connect 0x7f6d5806dfb0 2024-03-25T18:32:14.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.656+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f6d600546f0 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.656+0000 7f6d68ff9640 10 client.4516.objecter ms_dispatch 0x7f6d6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:14.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.656+0000 7f6d68ff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:14.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.656+0000 7f6d68ff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:14.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.656+0000 7f6d68ff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:32:14.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.657+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d60050030 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.657+0000 7f6d68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:14.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.657+0000 7f6d68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:14.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.906+0000 7f6d727fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.906+0000 7f6d727fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.906+0000 7f6d727fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:14.908 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.906+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d6c0627b0 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.933 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.931+0000 7f6d68ff9640 1 -- 172.21.15.102:0/949887987 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f6d60052130 con 0x7f6d6c0b4d60 2024-03-25T18:32:14.933 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.931+0000 7f6d68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:14.933 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.931+0000 7f6d68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:14.933 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:32:14.933 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:14.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 10 librados: watch_flush enter 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 10 librados: watch_flush exit 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 >> v1:172.21.15.146:6800/3848252714 conn(0x7f6d5806dfb0 legacy=0x7f6d58070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 10 monclient: shutdown 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 >> v1:172.21.15.146:6789/0 conn(0x7f6d6c0b4d60 legacy=0x7f6d6c139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 shutdown_connections 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 >> 172.21.15.102:0/949887987 conn(0x7f6d6c0617a0 msgr2=0x7f6d6c13c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 shutdown_connections 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.933+0000 7f6d727fb640 1 -- 172.21.15.102:0/949887987 wait complete. 2024-03-25T18:32:14.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:14.934+0000 7f6d727fb640 1 librados: shutdown 2024-03-25T18:32:14.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:21.755 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:21.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.870+0000 7f7f78b0e640 10 monclient: get_monmap_and_config 2024-03-25T18:32:21.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.870+0000 7f7f78b0e640 10 monclient: build_initial_monmap 2024-03-25T18:32:21.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.870+0000 7f7f78b0e640 10 monclient: monmap: 2024-03-25T18:32:21.872 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:21.872 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:21.873 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:32:21.872233+0000 2024-03-25T18:32:21.873 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:32:21.872233+0000 2024-03-25T18:32:21.873 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:21.873 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:21.873 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:21.873 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:32:21.873 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:32:21.873 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:21.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.871+0000 7f7f78b0e640 1 Processor -- start 2024-03-25T18:32:21.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.871+0000 7f7f78b0e640 1 -- start start 2024-03-25T18:32:21.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.871+0000 7f7f78b0e640 10 monclient: init 2024-03-25T18:32:21.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:21.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:21.874 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f7f740a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:21.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f7f7413f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:21.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f7f741430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:21.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f74142520 con 0x7f7f741430c0 2024-03-25T18:32:21.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f740b28a0 con 0x7f7f740a98f0 2024-03-25T18:32:21.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f740b2a30 con 0x7f7f7413f710 2024-03-25T18:32:21.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:21.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.872+0000 7f7f78b0e640 10 monclient(hunting): authenticate will time out at 7773.161541s 2024-03-25T18:32:21.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.877+0000 7f7f72d9d640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7f741430c0 0x7f7f7414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42436/0 (socket says 172.21.15.102:42436) 2024-03-25T18:32:21.878 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.877+0000 7f7f72d9d640 1 -- 172.21.15.102:0/2837480022 learned_addr learned my addr 172.21.15.102:0/2837480022 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:21.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.886+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847273072 0 0) 0x7f7f74142520 con 0x7f7f741430c0 2024-03-25T18:32:21.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.886+0000 7f7f7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.886+0000 7f7f7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.886+0000 7f7f7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.886+0000 7f7f7159a640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:32:21.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.886+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f5c003520 con 0x7f7f741430c0 2024-03-25T18:32:21.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.893+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471268336 0 0) 0x7f7f740b28a0 con 0x7f7f740a98f0 2024-03-25T18:32:21.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.893+0000 7f7f7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.893+0000 7f7f7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.893+0000 7f7f7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.893+0000 7f7f7159a640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:32:21.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.893+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f74142520 con 0x7f7f740a98f0 2024-03-25T18:32:21.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3154204637 0 0) 0x7f7f5c003520 con 0x7f7f741430c0 2024-03-25T18:32:21.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 >> v1:172.21.15.102:6790/0 conn(0x7f7f7413f710 legacy=0x7f7f741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 >> v1:172.21.15.146:6789/0 conn(0x7f7f740a98f0 legacy=0x7f7f740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 10 monclient: _finish_hunting 0 2024-03-25T18:32:21.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 1 monclient: found mon.noname-a 2024-03-25T18:32:21.903 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f740b44c0 con 0x7f7f741430c0 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 10 monclient: _finish_auth 0 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 10 monclient: _check_auth_tickets 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f78b0e640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7f6c001a10 con 0x7f7f741430c0 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f78b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f7159a640 10 monclient: dump: 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:21.904 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:21.905 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:21.905 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.905 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:21.905 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:21.905 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:21.905 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:21.905 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:21.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.901+0000 7f7f78b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:21.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.915+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f6c001b70 con 0x7f7f741430c0 2024-03-25T18:32:21.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f7159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f7159a640 1 -- 172.21.15.102:0/2837480022 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7f6c001f80 con 0x7f7f741430c0 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f7159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f7159a640 10 monclient: dump: 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:21.918 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:21.919 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:21.919 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:21.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f78b0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:21.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f78b0e640 4 set_mon_vals no callback set 2024-03-25T18:32:21.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f78b0e640 10 monclient: shutdown 2024-03-25T18:32:21.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f78b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:21.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2837480022 >> v1:172.21.15.102:6789/0 conn(0x7f7f741430c0 legacy=0x7f7f7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2837480022 shutdown_connections 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2837480022 >> 172.21.15.102:0/2837480022 conn(0x7f7f740b7ea0 msgr2=0x7f7f740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.916+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2837480022 shutdown_connections 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.917+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2837480022 wait complete. 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.917+0000 7f7f78b0e640 10 monclient: build_initial_monmap 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.917+0000 7f7f78b0e640 10 monclient: monmap: 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:32:21.921 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 librados: starting msgr at 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 librados: starting objecter 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 Processor -- start 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 -- start start 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 librados: setting wanted keys 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 librados: calling monclient init 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 10 monclient: init 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f7f740a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f7f7413f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:21.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f7f741430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:21.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f74139830 con 0x7f7f740a98f0 2024-03-25T18:32:21.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f741db7d0 con 0x7f7f741430c0 2024-03-25T18:32:21.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f741db410 con 0x7f7f7413f710 2024-03-25T18:32:21.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:21.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.918+0000 7f7f78b0e640 10 monclient(hunting): authenticate will time out at 7773.207858s 2024-03-25T18:32:21.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.921+0000 7f7f71d9b640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f7f7413f710 0x7f7f740b0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:43856/0 (socket says 172.21.15.102:43856) 2024-03-25T18:32:21.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.921+0000 7f7f71d9b640 1 -- 172.21.15.102:0/2529770333 learned_addr learned my addr 172.21.15.102:0/2529770333 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:21.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.921+0000 7f7f62ffd640 10 client.?.objecter ms_handle_connect 0x7f7f7413f710 2024-03-25T18:32:21.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.921+0000 7f7f62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.927+0000 7f7f62ffd640 10 client.?.objecter ms_handle_connect 0x7f7f740a98f0 2024-03-25T18:32:21.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.927+0000 7f7f62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.928+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896101724 0 0) 0x7f7f741db410 con 0x7f7f7413f710 2024-03-25T18:32:21.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.928+0000 7f7f62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.928+0000 7f7f62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.928+0000 7f7f62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.928+0000 7f7f62ffd640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:32:21.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.928+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f54004eb0 con 0x7f7f7413f710 2024-03-25T18:32:21.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.935+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3589225550 0 0) 0x7f7f54004eb0 con 0x7f7f7413f710 2024-03-25T18:32:21.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.935+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f741db410 con 0x7f7f7413f710 2024-03-25T18:32:21.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.935+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7f68001a30 con 0x7f7f7413f710 2024-03-25T18:32:21.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.935+0000 7f7f62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.935+0000 7f7f62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.935+0000 7f7f62ffd640 10 monclient(hunting): dump: 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:21.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.935+0000 7f7f62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:21.944 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.943+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395188735 0 0) 0x7f7f74139830 con 0x7f7f740a98f0 2024-03-25T18:32:21.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.943+0000 7f7f62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:32:21.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.943+0000 7f7f62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:21.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.943+0000 7f7f62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:21.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.943+0000 7f7f62ffd640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:32:21.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.943+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f54004eb0 con 0x7f7f740a98f0 2024-03-25T18:32:21.945 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.944+0000 7f7f62ffd640 10 client.?.objecter ms_handle_connect 0x7f7f741430c0 2024-03-25T18:32:21.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.944+0000 7f7f62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:21.949 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3315277762 0 0) 0x7f7f741db410 con 0x7f7f7413f710 2024-03-25T18:32:21.949 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 >> v1:172.21.15.146:6789/0 conn(0x7f7f741430c0 legacy=0x7f7f741d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 >> v1:172.21.15.102:6789/0 conn(0x7f7f740a98f0 legacy=0x7f7f741dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 1 monclient: found mon.c 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f741db5a0 con 0x7f7f7413f710 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 10 monclient: _finish_auth 0 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 5 monclient: authenticate success, global_id 4496 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 10 monclient: _renew_subs 2024-03-25T18:32:21.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:32:21.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f740b1480 con 0x7f7f7413f710 2024-03-25T18:32:21.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:21.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 10 monclient: _renew_subs 2024-03-25T18:32:21.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:32:21.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f740b0b10 con 0x7f7f7413f710 2024-03-25T18:32:21.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.947+0000 7f7f78b0e640 1 librados: init done 2024-03-25T18:32:21.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.949+0000 7f7f78b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:21.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.949+0000 7f7f78b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:21.952 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.949+0000 7f7f78b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:32:21.952 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:21.949+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f740627b0 con 0x7f7f7413f710 2024-03-25T18:32:22.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.239+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f68001cf0 con 0x7f7f7413f710 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.239+0000 7f7f62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.239+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7f68003a60 con 0x7f7f7413f710 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.239+0000 7f7f62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.239+0000 7f7f62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.239+0000 7f7f62ffd640 10 monclient: dump: 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:22.242 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:22.243 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:22.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.239+0000 7f7f70d99640 4 set_mon_vals no callback set 2024-03-25T18:32:22.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.240+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f7f68003c80 con 0x7f7f7413f710 2024-03-25T18:32:22.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.241+0000 7f7f7259c640 10 monclient: get_auth_request con 0x7f7f5406dd00 auth_method 0 2024-03-25T18:32:22.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.241+0000 7f7f62ffd640 10 client.4496.objecter ms_handle_connect 0x7f7f5406dd00 2024-03-25T18:32:22.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.247+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f7f68054930 con 0x7f7f7413f710 2024-03-25T18:32:22.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.247+0000 7f7f62ffd640 10 client.4496.objecter ms_dispatch 0x7f7f74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:22.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.247+0000 7f7f62ffd640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:22.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.247+0000 7f7f62ffd640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:22.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.247+0000 7f7f62ffd640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T18:32:22.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.247+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f68050270 con 0x7f7f7413f710 2024-03-25T18:32:22.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.247+0000 7f7f62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:22.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.247+0000 7f7f62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:22.473 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.471+0000 7f7f78b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:22.473 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.471+0000 7f7f78b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:22.473 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.471+0000 7f7f78b0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:32:22.473 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.471+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f741db960 con 0x7f7f7413f710 2024-03-25T18:32:22.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.498+0000 7f7f62ffd640 1 -- 172.21.15.102:0/2529770333 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f7f68052370 con 0x7f7f7413f710 2024-03-25T18:32:22.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.498+0000 7f7f62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:22.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.498+0000 7f7f62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:22.501 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:32:22.501 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.500+0000 7f7f78b0e640 10 librados: watch_flush enter 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 10 librados: watch_flush exit 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 10 client.4496.objecter successfully canceled tick 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 >> v1:172.21.15.146:6800/3848252714 conn(0x7f7f5406dd00 legacy=0x7f7f540701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 10 monclient: shutdown 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 >> v1:172.21.15.102:6790/0 conn(0x7f7f7413f710 legacy=0x7f7f740b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:22.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 shutdown_connections 2024-03-25T18:32:22.504 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 >> 172.21.15.102:0/2529770333 conn(0x7f7f740b7ea0 msgr2=0x7f7f74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:22.504 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 shutdown_connections 2024-03-25T18:32:22.504 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 1 -- 172.21.15.102:0/2529770333 wait complete. 2024-03-25T18:32:22.504 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:22.501+0000 7f7f78b0e640 1 librados: shutdown 2024-03-25T18:32:22.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:23.947 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:40.182 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:24.000 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:40.236 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:24.059 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:06:40.295 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:24.062 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:40.275 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:24.068 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:40.304 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:24.073 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:40.286 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:24.076 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:40.312 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:24.104 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:06:40.317 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:24.108 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:06:40.344 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:29.321 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.424+0000 7f26b2f96640 10 monclient: get_monmap_and_config 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.424+0000 7f26b2f96640 10 monclient: build_initial_monmap 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.424+0000 7f26b2f96640 10 monclient: monmap: 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:32:29.425887+0000 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:32:29.425887+0000 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:29.426 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:32:29.427 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:32:29.427 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.424+0000 7f26b2f96640 1 Processor -- start 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 1 -- start start 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 10 monclient: init 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 10 monclient(hunting): picked mon.noname-c con 0x7f26ac0b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 10 monclient(hunting): picked mon.noname-a con 0x7f26ac0b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 10 monclient(hunting): picked mon.noname-b con 0x7f26ac14be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac0af940 con 0x7f26ac0b1730 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac0b0a70 con 0x7f26ac0b4d60 2024-03-25T18:32:29.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac0b0d50 con 0x7f26ac14be90 2024-03-25T18:32:29.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:29.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.425+0000 7f26b2f96640 10 monclient(hunting): authenticate will time out at 7780.714543s 2024-03-25T18:32:29.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.443+0000 7f26b0d32640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f26ac0b4d60 0x7f26ac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:36098/0 (socket says 172.21.15.102:36098) 2024-03-25T18:32:29.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.443+0000 7f26b0d32640 1 -- 172.21.15.102:0/4232446825 learned_addr learned my addr 172.21.15.102:0/4232446825 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:29.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.477+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049684723 0 0) 0x7f26ac0b0a70 con 0x7f26ac0b4d60 2024-03-25T18:32:29.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.477+0000 7f26ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.477+0000 7f26ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.477+0000 7f26ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.477+0000 7f26ab7fe640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:32:29.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.477+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f269c003520 con 0x7f26ac0b4d60 2024-03-25T18:32:29.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.485+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652237819 0 0) 0x7f26ac0b0d50 con 0x7f26ac14be90 2024-03-25T18:32:29.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.485+0000 7f26ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.485+0000 7f26ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.485+0000 7f26ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.485+0000 7f26ab7fe640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:32:29.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.486+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac0b0a70 con 0x7f26ac14be90 2024-03-25T18:32:29.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.487+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509133183 0 0) 0x7f26ac0af940 con 0x7f26ac0b1730 2024-03-25T18:32:29.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.487+0000 7f26ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.487+0000 7f26ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.487+0000 7f26ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.487+0000 7f26ab7fe640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:32:29.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.487+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac0b0d50 con 0x7f26ac0b1730 2024-03-25T18:32:29.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 624694132 0 0) 0x7f26ac0b0a70 con 0x7f26ac14be90 2024-03-25T18:32:29.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 >> v1:172.21.15.146:6789/0 conn(0x7f26ac0b4d60 legacy=0x7f26ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 >> v1:172.21.15.102:6789/0 conn(0x7f26ac0b1730 legacy=0x7f26ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:29.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 1 monclient: found mon.noname-b 2024-03-25T18:32:29.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:29.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac0b1030 con 0x7f26ac14be90 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f26a40019f0 con 0x7f26ac14be90 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26ab7fe640 10 monclient: dump: 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:29.500 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:29.501 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:29.501 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:29.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26b2f96640 5 monclient: authenticate success, global_id 4499 2024-03-25T18:32:29.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.497+0000 7f26b2f96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:29.532 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.530+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26a4001b50 con 0x7f26ac14be90 2024-03-25T18:32:29.532 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.530+0000 7f26ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:29.532 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.530+0000 7f26ab7fe640 1 -- 172.21.15.102:0/4232446825 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f26a4001f60 con 0x7f26ac14be90 2024-03-25T18:32:29.532 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.530+0000 7f26ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.532 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.530+0000 7f26ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.530+0000 7f26ab7fe640 10 monclient: dump: 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.530+0000 7f26b2f96640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:29.533 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.530+0000 7f26b2f96640 4 set_mon_vals no callback set 2024-03-25T18:32:29.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.531+0000 7f26b2f96640 10 monclient: shutdown 2024-03-25T18:32:29.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.531+0000 7f26b2f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:29.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.531+0000 7f26b2f96640 1 -- 172.21.15.102:0/4232446825 >> v1:172.21.15.102:6790/0 conn(0x7f26ac14be90 legacy=0x7f26ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.531+0000 7f26b2f96640 1 -- 172.21.15.102:0/4232446825 shutdown_connections 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.531+0000 7f26b2f96640 1 -- 172.21.15.102:0/4232446825 >> 172.21.15.102:0/4232446825 conn(0x7f26ac0617a0 msgr2=0x7f26ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.531+0000 7f26b2f96640 1 -- 172.21.15.102:0/4232446825 shutdown_connections 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.531+0000 7f26b2f96640 1 -- 172.21.15.102:0/4232446825 wait complete. 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 10 monclient: build_initial_monmap 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 10 monclient: monmap: 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:32:29.536 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 1 librados: starting msgr at 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 1 librados: starting objecter 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 1 Processor -- start 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 1 -- start start 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 1 librados: setting wanted keys 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 1 librados: calling monclient init 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.532+0000 7f26b2f96640 10 monclient: init 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 10 monclient(hunting): picked mon.noname-a con 0x7f26ac0b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 10 monclient(hunting): picked mon.noname-c con 0x7f26ac0b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 10 monclient(hunting): picked mon.noname-b con 0x7f26ac14be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac13f0c0 con 0x7f26ac0b1730 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac140380 con 0x7f26ac14be90 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ac140660 con 0x7f26ac0b4d60 2024-03-25T18:32:29.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:29.538 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.533+0000 7f26b2f96640 10 monclient(hunting): authenticate will time out at 7780.822880s 2024-03-25T18:32:29.545 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.543+0000 7f26b0d32640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f26ac0b1730 0x7f26ac131e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47230/0 (socket says 172.21.15.102:47230) 2024-03-25T18:32:29.545 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.543+0000 7f26b0d32640 1 -- 172.21.15.102:0/1739905320 learned_addr learned my addr 172.21.15.102:0/1739905320 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:29.546 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.545+0000 7f26a97fa640 10 client.?.objecter ms_handle_connect 0x7f26ac0b1730 2024-03-25T18:32:29.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.545+0000 7f26a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:29.553 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.550+0000 7f26a97fa640 10 client.?.objecter ms_handle_connect 0x7f26ac14be90 2024-03-25T18:32:29.553 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.550+0000 7f26a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:29.554 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.552+0000 7f26a97fa640 10 client.?.objecter ms_handle_connect 0x7f26ac0b4d60 2024-03-25T18:32:29.554 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.552+0000 7f26a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:29.555 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.554+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378215531 0 0) 0x7f26ac13f0c0 con 0x7f26ac0b1730 2024-03-25T18:32:29.555 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.554+0000 7f26a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.554+0000 7f26a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.554+0000 7f26a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.554+0000 7f26a97fa640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:32:29.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.554+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2690004eb0 con 0x7f26ac0b1730 2024-03-25T18:32:29.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.560+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256305368 0 0) 0x7f26ac140380 con 0x7f26ac14be90 2024-03-25T18:32:29.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.560+0000 7f26a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.560+0000 7f26a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.560+0000 7f26a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.560+0000 7f26a97fa640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:32:29.562 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.560+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac13f0c0 con 0x7f26ac14be90 2024-03-25T18:32:29.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.567+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100460920 0 0) 0x7f26ac140660 con 0x7f26ac0b4d60 2024-03-25T18:32:29.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.567+0000 7f26a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:29.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.567+0000 7f26a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:29.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.567+0000 7f26a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:29.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.567+0000 7f26a97fa640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:32:29.569 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.567+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac140380 con 0x7f26ac0b4d60 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.569+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4181481978 0 0) 0x7f2690004eb0 con 0x7f26ac0b1730 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.569+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26ac140660 con 0x7f26ac0b1730 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.569+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f26a0001950 con 0x7f26ac0b1730 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.569+0000 7f26a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.569+0000 7f26a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.569+0000 7f26a97fa640 10 monclient(hunting): dump: 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.571 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.572 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:29.572 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:29.572 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:29.572 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:29.572 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:29.572 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.569+0000 7f26a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:29.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.577+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3421634102 0 0) 0x7f26ac13f0c0 con 0x7f26ac14be90 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.577+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2690004eb0 con 0x7f26ac14be90 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.577+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f26a4001e60 con 0x7f26ac14be90 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.577+0000 7f26a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.577+0000 7f26a97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.577+0000 7f26a97fa640 10 monclient(hunting): dump: 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:29.579 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1464100627 0 0) 0x7f26ac140660 con 0x7f26ac0b1730 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 >> v1:172.21.15.102:6790/0 conn(0x7f26ac0b4d60 legacy=0x7f26ac142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 >> v1:172.21.15.146:6789/0 conn(0x7f26ac14be90 legacy=0x7f26ac1e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 1 monclient: found mon.a 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac132520 con 0x7f26ac0b1730 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:29.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:32:29.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 10 monclient: _renew_subs 2024-03-25T18:32:29.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:29.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ac132770 con 0x7f26ac0b1730 2024-03-25T18:32:29.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:29.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 10 monclient: _renew_subs 2024-03-25T18:32:29.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:29.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ac13e790 con 0x7f26ac0b1730 2024-03-25T18:32:29.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.580+0000 7f26b2f96640 1 librados: init done 2024-03-25T18:32:29.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.581+0000 7f26b2f96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.581+0000 7f26b2f96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.581+0000 7f26b2f96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:29.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.581+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ac142110 con 0x7f26ac0b1730 2024-03-25T18:32:29.887 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.885+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26a00047e0 con 0x7f26ac0b1730 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.885+0000 7f26a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.885+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f26a0004af0 con 0x7f26ac0b1730 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.885+0000 7f26a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.885+0000 7f26a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.885+0000 7f26a97fa640 10 monclient: dump: 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:29.888 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:29.889 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:29.889 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:29.889 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.885+0000 7f26aaffd640 4 set_mon_vals no callback set 2024-03-25T18:32:29.889 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.886+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f26a001aac0 con 0x7f26ac0b1730 2024-03-25T18:32:29.889 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.887+0000 7f26abfff640 10 monclient: get_auth_request con 0x7f269006df20 auth_method 0 2024-03-25T18:32:29.889 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.887+0000 7f26a97fa640 10 client.4527.objecter ms_handle_connect 0x7f269006df20 2024-03-25T18:32:29.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.893+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f26a0054850 con 0x7f26ac0b1730 2024-03-25T18:32:29.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.893+0000 7f26a97fa640 10 client.4527.objecter ms_dispatch 0x7f26ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:29.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.893+0000 7f26a97fa640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:29.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.893+0000 7f26a97fa640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:29.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.893+0000 7f26a97fa640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T18:32:29.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.893+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26a0050190 con 0x7f26ac0b1730 2024-03-25T18:32:29.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.893+0000 7f26a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:29.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:29.893+0000 7f26a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:30.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.146+0000 7f26b2f96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:30.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.146+0000 7f26b2f96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:30.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.146+0000 7f26b2f96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:30.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.146+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26ac1407f0 con 0x7f26ac0b1730 2024-03-25T18:32:30.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.179+0000 7f26a97fa640 1 -- 172.21.15.102:0/1739905320 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f26a0052290 con 0x7f26ac0b1730 2024-03-25T18:32:30.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.179+0000 7f26a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:30.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.179+0000 7f26a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:30.182 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:32:30.182 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:30.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 10 librados: watch_flush enter 2024-03-25T18:32:30.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 10 librados: watch_flush exit 2024-03-25T18:32:30.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 10 client.4527.objecter successfully canceled tick 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 >> v1:172.21.15.146:6800/3848252714 conn(0x7f269006df20 legacy=0x7f26900703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 10 monclient: shutdown 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 >> v1:172.21.15.102:6789/0 conn(0x7f26ac0b1730 legacy=0x7f26ac131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 shutdown_connections 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 >> 172.21.15.102:0/1739905320 conn(0x7f26ac0617a0 msgr2=0x7f26ac137bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 shutdown_connections 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 1 -- 172.21.15.102:0/1739905320 wait complete. 2024-03-25T18:32:30.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:30.182+0000 7f26b2f96640 1 librados: shutdown 2024-03-25T18:32:30.197 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:37.002 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.119+0000 7f92b314b640 10 monclient: get_monmap_and_config 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.119+0000 7f92b314b640 10 monclient: build_initial_monmap 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.119+0000 7f92b314b640 10 monclient: monmap: 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:32:37.119895+0000 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:32:37.119895+0000 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:32:37.120 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:37.121 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.120+0000 7f92b314b640 1 Processor -- start 2024-03-25T18:32:37.121 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.120+0000 7f92b314b640 1 -- start start 2024-03-25T18:32:37.121 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.120+0000 7f92b314b640 10 monclient: init 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 10 monclient(hunting): picked mon.noname-b con 0x7f92ac0a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 10 monclient(hunting): picked mon.noname-a con 0x7f92ac13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 10 monclient(hunting): picked mon.noname-c con 0x7f92ac1430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac142520 con 0x7f92ac13f710 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0b28a0 con 0x7f92ac1430c0 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0b2a30 con 0x7f92ac0a98f0 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:37.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.121+0000 7f92b314b640 10 monclient(hunting): authenticate will time out at 7788.409135s 2024-03-25T18:32:37.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.124+0000 7f92b16e8640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f92ac1430c0 0x7f92ac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:44124/0 (socket says 172.21.15.102:44124) 2024-03-25T18:32:37.124 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.124+0000 7f92b16e8640 1 -- 172.21.15.102:0/1265770990 learned_addr learned my addr 172.21.15.102:0/1265770990 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:37.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.143+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136676262 0 0) 0x7f92ac0b28a0 con 0x7f92ac1430c0 2024-03-25T18:32:37.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.143+0000 7f92ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:37.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.143+0000 7f92ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:37.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.143+0000 7f92ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:37.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.143+0000 7f92ab7fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:32:37.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.143+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929c003520 con 0x7f92ac1430c0 2024-03-25T18:32:37.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.152+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545424527 0 0) 0x7f92ac0b2a30 con 0x7f92ac0a98f0 2024-03-25T18:32:37.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.153+0000 7f92ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:37.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.153+0000 7f92ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:37.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.153+0000 7f92ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:37.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.153+0000 7f92ab7fe640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:32:37.153 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.153+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac0b28a0 con 0x7f92ac0a98f0 2024-03-25T18:32:37.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.161+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1965683072 0 0) 0x7f929c003520 con 0x7f92ac1430c0 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 >> v1:172.21.15.102:6790/0 conn(0x7f92ac0a98f0 legacy=0x7f92ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 >> v1:172.21.15.102:6789/0 conn(0x7f92ac13f710 legacy=0x7f92ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 1 monclient: found mon.noname-c 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac0b44c0 con 0x7f92ac1430c0 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f92a4001a10 con 0x7f92ac1430c0 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92ab7fe640 10 monclient: dump: 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:37.163 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92b314b640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:32:37.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.162+0000 7f92b314b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:37.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.183+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a4001b70 con 0x7f92ac1430c0 2024-03-25T18:32:37.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.183+0000 7f92ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:37.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.183+0000 7f92ab7fe640 1 -- 172.21.15.102:0/1265770990 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f92a4001f80 con 0x7f92ac1430c0 2024-03-25T18:32:37.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.183+0000 7f92b314b640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:37.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.183+0000 7f92b314b640 4 set_mon_vals no callback set 2024-03-25T18:32:37.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.183+0000 7f92ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:37.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.183+0000 7f92ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:37.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.183+0000 7f92ab7fe640 10 monclient: dump: 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:37.185 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:37.186 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.184+0000 7f92b314b640 10 monclient: shutdown 2024-03-25T18:32:37.186 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.184+0000 7f92b314b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.184+0000 7f92b314b640 1 -- 172.21.15.102:0/1265770990 >> v1:172.21.15.146:6789/0 conn(0x7f92ac1430c0 legacy=0x7f92ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.184+0000 7f92b314b640 1 -- 172.21.15.102:0/1265770990 shutdown_connections 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.184+0000 7f92b314b640 1 -- 172.21.15.102:0/1265770990 >> 172.21.15.102:0/1265770990 conn(0x7f92ac0b7ea0 msgr2=0x7f92ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.184+0000 7f92b314b640 1 -- 172.21.15.102:0/1265770990 shutdown_connections 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.184+0000 7f92b314b640 1 -- 172.21.15.102:0/1265770990 wait complete. 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.185+0000 7f92b314b640 10 monclient: build_initial_monmap 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.185+0000 7f92b314b640 10 monclient: monmap: 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:32:37.187 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:32:37.188 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:37.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 1 librados: starting msgr at 2024-03-25T18:32:37.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 1 librados: starting objecter 2024-03-25T18:32:37.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 1 Processor -- start 2024-03-25T18:32:37.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 1 -- start start 2024-03-25T18:32:37.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 1 librados: setting wanted keys 2024-03-25T18:32:37.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 1 librados: calling monclient init 2024-03-25T18:32:37.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 10 monclient: init 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.186+0000 7f92b314b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.187+0000 7f92b314b640 10 monclient(hunting): picked mon.noname-a con 0x7f92ac0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.187+0000 7f92b314b640 10 monclient(hunting): picked mon.noname-b con 0x7f92ac13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.187+0000 7f92b314b640 10 monclient(hunting): picked mon.noname-c con 0x7f92ac1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.187+0000 7f92b314b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0b7a80 con 0x7f92ac0a98f0 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.187+0000 7f92b314b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0b77a0 con 0x7f92ac13f710 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.187+0000 7f92b314b640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac136bb0 con 0x7f92ac1430c0 2024-03-25T18:32:37.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.187+0000 7f92b314b640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:37.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.187+0000 7f92b314b640 10 monclient(hunting): authenticate will time out at 7788.475178s 2024-03-25T18:32:37.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.196+0000 7f92b0ee7640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f92ac0a98f0 0x7f92ac137590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46290/0 (socket says 172.21.15.102:46290) 2024-03-25T18:32:37.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.197+0000 7f92b0ee7640 1 -- 172.21.15.102:0/2984994437 learned_addr learned my addr 172.21.15.102:0/2984994437 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:37.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.199+0000 7f92a97fa640 10 client.?.objecter ms_handle_connect 0x7f92ac13f710 2024-03-25T18:32:37.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.199+0000 7f92a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:37.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.200+0000 7f92a97fa640 10 client.?.objecter ms_handle_connect 0x7f92ac0a98f0 2024-03-25T18:32:37.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.200+0000 7f92a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:37.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.208+0000 7f92a97fa640 10 client.?.objecter ms_handle_connect 0x7f92ac1430c0 2024-03-25T18:32:37.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.208+0000 7f92a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:37.214 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.214+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402205274 0 0) 0x7f92ac0b77a0 con 0x7f92ac13f710 2024-03-25T18:32:37.214 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.214+0000 7f92a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:37.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.214+0000 7f92a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:37.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.214+0000 7f92a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:37.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.214+0000 7f92a97fa640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:32:37.215 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.214+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9290004eb0 con 0x7f92ac13f710 2024-03-25T18:32:37.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.218+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911759353 0 0) 0x7f92ac0b7a80 con 0x7f92ac0a98f0 2024-03-25T18:32:37.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.218+0000 7f92a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:37.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.218+0000 7f92a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:37.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.218+0000 7f92a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:37.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.218+0000 7f92a97fa640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:32:37.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.218+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac0b77a0 con 0x7f92ac0a98f0 2024-03-25T18:32:37.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.222+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245343256 0 0) 0x7f92ac136bb0 con 0x7f92ac1430c0 2024-03-25T18:32:37.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.222+0000 7f92a97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:37.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.222+0000 7f92a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:37.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.222+0000 7f92a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:37.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.222+0000 7f92a97fa640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:32:37.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.222+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac0b7a80 con 0x7f92ac1430c0 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.233+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3346421182 0 0) 0x7f9290004eb0 con 0x7f92ac13f710 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.233+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92ac136bb0 con 0x7f92ac13f710 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.233+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f92980018c0 con 0x7f92ac13f710 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.233+0000 7f92a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.233+0000 7f92a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.233+0000 7f92a97fa640 10 monclient(hunting): dump: 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.234 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:37.235 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:37.235 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:37.235 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:37.235 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:37.235 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:37.235 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.233+0000 7f92a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.237+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2098673398 0 0) 0x7f92ac0b77a0 con 0x7f92ac0a98f0 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.237+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9290004eb0 con 0x7f92ac0a98f0 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.237+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f92a00019f0 con 0x7f92ac0a98f0 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.237+0000 7f92a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.237+0000 7f92a97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.237+0000 7f92a97fa640 10 monclient(hunting): dump: 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.238 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:37.239 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:37.239 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:37.239 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:37.239 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:37.239 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:37.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.243+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2519796226 0 0) 0x7f92ac0b7a80 con 0x7f92ac1430c0 2024-03-25T18:32:37.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.243+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92ac0b77a0 con 0x7f92ac1430c0 2024-03-25T18:32:37.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.243+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f92a4001e80 con 0x7f92ac1430c0 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.243+0000 7f92a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.243+0000 7f92a97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.243+0000 7f92a97fa640 10 monclient(hunting): dump: 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:37.244 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 78293494 0 0) 0x7f92ac136bb0 con 0x7f92ac13f710 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 >> v1:172.21.15.102:6790/0 conn(0x7f92ac1430c0 legacy=0x7f92ac1dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 >> v1:172.21.15.102:6789/0 conn(0x7f92ac0a98f0 legacy=0x7f92ac137590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 1 monclient: found mon.b 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac1dc150 con 0x7f92ac13f710 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:37.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 10 monclient: _renew_subs 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac136ef0 con 0x7f92ac13f710 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 10 monclient: _renew_subs 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac137ca0 con 0x7f92ac13f710 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.248+0000 7f92b314b640 1 librados: init done 2024-03-25T18:32:37.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.249+0000 7f92b314b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:37.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.249+0000 7f92b314b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:37.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.249+0000 7f92b314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:37.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.249+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac13b210 con 0x7f92ac13f710 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.554+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92980035f0 con 0x7f92ac13f710 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.554+0000 7f92a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.554+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9298003760 con 0x7f92ac13f710 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.554+0000 7f92a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.554+0000 7f92a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.554+0000 7f92a97fa640 10 monclient: dump: 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:37.556 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:37.557 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:37.557 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:37.557 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:37.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.554+0000 7f92aaffd640 4 set_mon_vals no callback set 2024-03-25T18:32:37.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.556+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f9298019730 con 0x7f92ac13f710 2024-03-25T18:32:37.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.556+0000 7f92b0ee7640 10 monclient: get_auth_request con 0x7f929006e0b0 auth_method 0 2024-03-25T18:32:37.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.556+0000 7f92a97fa640 10 client.4534.objecter ms_handle_connect 0x7f929006e0b0 2024-03-25T18:32:37.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.559+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f9298053410 con 0x7f92ac13f710 2024-03-25T18:32:37.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.559+0000 7f92a97fa640 10 client.4534.objecter ms_dispatch 0x7f92ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:37.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.559+0000 7f92a97fa640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:37.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.559+0000 7f92a97fa640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:37.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.559+0000 7f92a97fa640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T18:32:37.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.560+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f929804ef20 con 0x7f92ac13f710 2024-03-25T18:32:37.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.560+0000 7f92a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:37.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.560+0000 7f92a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:37.790 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.790+0000 7f92b314b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.790+0000 7f92b314b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.790+0000 7f92b314b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:37.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.790+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92ac0b7c80 con 0x7f92ac13f710 2024-03-25T18:32:37.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.822+0000 7f92a97fa640 1 -- 172.21.15.102:0/2984994437 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f9298050e50 con 0x7f92ac13f710 2024-03-25T18:32:37.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.822+0000 7f92a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:37.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.822+0000 7f92a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:37.823 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:32:37.824 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:37.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.825+0000 7f92b314b640 10 librados: watch_flush enter 2024-03-25T18:32:37.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.825+0000 7f92b314b640 10 librados: watch_flush exit 2024-03-25T18:32:37.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.825+0000 7f92b314b640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T18:32:37.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.825+0000 7f92b314b640 10 client.4534.objecter successfully canceled tick 2024-03-25T18:32:37.826 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.825+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 >> v1:172.21.15.146:6800/3848252714 conn(0x7f929006e0b0 legacy=0x7f9290070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.825+0000 7f92b314b640 10 monclient: shutdown 2024-03-25T18:32:37.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.825+0000 7f92b314b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:37.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.825+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 >> v1:172.21.15.146:6789/0 conn(0x7f92ac13f710 legacy=0x7f92ac1db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:37.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.826+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 shutdown_connections 2024-03-25T18:32:37.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.826+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 >> 172.21.15.102:0/2984994437 conn(0x7f92ac0b7ea0 msgr2=0x7f92ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:37.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.826+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 shutdown_connections 2024-03-25T18:32:37.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.826+0000 7f92b314b640 1 -- 172.21.15.102:0/2984994437 wait complete. 2024-03-25T18:32:37.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:37.826+0000 7f92b314b640 1 librados: shutdown 2024-03-25T18:32:37.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:44.027 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:00.263 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.089 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:00.325 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.124 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:00.359 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.125 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:00.361 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.127 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:00.340 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.133 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:00.369 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.138 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:00.351 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.157 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:00.393 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.173 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:00.386 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:32:44.647 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:44.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.762+0000 7f6503065640 10 monclient: get_monmap_and_config 2024-03-25T18:32:44.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.762+0000 7f6503065640 10 monclient: build_initial_monmap 2024-03-25T18:32:44.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.762+0000 7f6503065640 10 monclient: monmap: 2024-03-25T18:32:44.763 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:44.763 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:44.763 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:32:44.763036+0000 2024-03-25T18:32:44.764 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:32:44.763036+0000 2024-03-25T18:32:44.764 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:44.764 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:44.764 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:44.764 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:32:44.764 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:32:44.764 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:44.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.763+0000 7f6503065640 1 Processor -- start 2024-03-25T18:32:44.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.763+0000 7f6503065640 1 -- start start 2024-03-25T18:32:44.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.763+0000 7f6503065640 10 monclient: init 2024-03-25T18:32:44.765 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:44.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:44.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 10 monclient(hunting): picked mon.noname-c con 0x7f64fc0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:44.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 10 monclient(hunting): picked mon.noname-b con 0x7f64fc13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:44.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 10 monclient(hunting): picked mon.noname-a con 0x7f64fc1430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:44.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc142520 con 0x7f64fc1430c0 2024-03-25T18:32:44.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc0b28a0 con 0x7f64fc0a98f0 2024-03-25T18:32:44.766 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc0b2a30 con 0x7f64fc13f710 2024-03-25T18:32:44.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:44.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.764+0000 7f6503065640 10 monclient(hunting): authenticate will time out at 7796.052777s 2024-03-25T18:32:44.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.778+0000 7f6501602640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f64fc1430c0 0x7f64fc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58314/0 (socket says 172.21.15.102:58314) 2024-03-25T18:32:44.779 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.778+0000 7f6501602640 1 -- 172.21.15.102:0/2100468123 learned_addr learned my addr 172.21.15.102:0/2100468123 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:44.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.791+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257980703 0 0) 0x7f64fc0b28a0 con 0x7f64fc0a98f0 2024-03-25T18:32:44.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.791+0000 7f64fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.791+0000 7f64fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.791+0000 7f64fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.791+0000 7f64fb7fe640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:32:44.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.791+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ec003520 con 0x7f64fc0a98f0 2024-03-25T18:32:44.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.792+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608587706 0 0) 0x7f64fc142520 con 0x7f64fc1430c0 2024-03-25T18:32:44.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.792+0000 7f64fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.792+0000 7f64fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.792+0000 7f64fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.792+0000 7f64fb7fe640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:32:44.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.792+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64fc0b28a0 con 0x7f64fc1430c0 2024-03-25T18:32:44.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3169797682 0 0) 0x7f64fc0b28a0 con 0x7f64fc1430c0 2024-03-25T18:32:44.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 >> v1:172.21.15.102:6790/0 conn(0x7f64fc13f710 legacy=0x7f64fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 >> v1:172.21.15.146:6789/0 conn(0x7f64fc0a98f0 legacy=0x7f64fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:32:44.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 1 monclient: found mon.noname-a 2024-03-25T18:32:44.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:44.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:32:44.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc0b44c0 con 0x7f64fc1430c0 2024-03-25T18:32:44.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:32:44.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:32:44.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f64fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:44.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f6503065640 5 monclient: authenticate success, global_id 4536 2024-03-25T18:32:44.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.802+0000 7f6503065640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:44.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.804+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f64f40019b0 con 0x7f64fc1430c0 2024-03-25T18:32:44.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.804+0000 7f64fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.804+0000 7f64fb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:44.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.804+0000 7f64fb7fe640 10 monclient: dump: 2024-03-25T18:32:44.807 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:44.807 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:44.807 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:44.808 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:44.808 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.808 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:44.808 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:44.808 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:44.809 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:44.809 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:44.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.804+0000 7f6503065640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.830+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64f4001c40 con 0x7f64fc1430c0 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.830+0000 7f64fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.830+0000 7f64fb7fe640 1 -- 172.21.15.102:0/2100468123 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f64f4001fd0 con 0x7f64fc1430c0 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.830+0000 7f64fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.830+0000 7f64fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.830+0000 7f64fb7fe640 10 monclient: dump: 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:44.831 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:44.832 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.832 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:44.832 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:44.832 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:44.832 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:44.832 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:44.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.830+0000 7f6503065640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:44.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.830+0000 7f6503065640 4 set_mon_vals no callback set 2024-03-25T18:32:44.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 10 monclient: shutdown 2024-03-25T18:32:44.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:44.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 1 -- 172.21.15.102:0/2100468123 >> v1:172.21.15.102:6789/0 conn(0x7f64fc1430c0 legacy=0x7f64fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 1 -- 172.21.15.102:0/2100468123 shutdown_connections 2024-03-25T18:32:44.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 1 -- 172.21.15.102:0/2100468123 >> 172.21.15.102:0/2100468123 conn(0x7f64fc0b7ea0 msgr2=0x7f64fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:44.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 1 -- 172.21.15.102:0/2100468123 shutdown_connections 2024-03-25T18:32:44.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 1 -- 172.21.15.102:0/2100468123 wait complete. 2024-03-25T18:32:44.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 10 monclient: build_initial_monmap 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.831+0000 7f6503065640 10 monclient: monmap: 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:44.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.832+0000 7f6503065640 1 librados: starting msgr at 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.832+0000 7f6503065640 1 librados: starting objecter 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.832+0000 7f6503065640 1 Processor -- start 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.832+0000 7f6503065640 1 -- start start 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.832+0000 7f6503065640 1 librados: setting wanted keys 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.832+0000 7f6503065640 1 librados: calling monclient init 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.832+0000 7f6503065640 10 monclient: init 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:32:44.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 10 monclient(hunting): picked mon.noname-a con 0x7f64fc0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:44.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 10 monclient(hunting): picked mon.noname-c con 0x7f64fc13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:44.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 10 monclient(hunting): picked mon.noname-b con 0x7f64fc1430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:44.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc137f90 con 0x7f64fc0a98f0 2024-03-25T18:32:44.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc0abb70 con 0x7f64fc1430c0 2024-03-25T18:32:44.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc137c90 con 0x7f64fc13f710 2024-03-25T18:32:44.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:44.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.833+0000 7f6503065640 10 monclient(hunting): authenticate will time out at 7796.121350s 2024-03-25T18:32:44.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.837+0000 7f6501602640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f64fc1430c0 0x7f64fc1e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:44794/0 (socket says 172.21.15.102:44794) 2024-03-25T18:32:44.839 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.837+0000 7f6501602640 1 -- 172.21.15.102:0/3078769570 learned_addr learned my addr 172.21.15.102:0/3078769570 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:44.839 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.838+0000 7f64f97fa640 10 client.?.objecter ms_handle_connect 0x7f64fc1430c0 2024-03-25T18:32:44.839 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.838+0000 7f64f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.842+0000 7f64f97fa640 10 client.?.objecter ms_handle_connect 0x7f64fc13f710 2024-03-25T18:32:44.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.842+0000 7f64f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.843+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815364467 0 0) 0x7f64fc0abb70 con 0x7f64fc1430c0 2024-03-25T18:32:44.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.843+0000 7f64f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.843+0000 7f64f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.843+0000 7f64f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.843+0000 7f64f97fa640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:32:44.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.843+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e0004eb0 con 0x7f64fc1430c0 2024-03-25T18:32:44.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.847+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597270805 0 0) 0x7f64fc137c90 con 0x7f64fc13f710 2024-03-25T18:32:44.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.847+0000 7f64f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:32:44.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.847+0000 7f64f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:44.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.847+0000 7f64f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:44.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.847+0000 7f64f97fa640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:32:44.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.847+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64fc0abb70 con 0x7f64fc13f710 2024-03-25T18:32:44.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.848+0000 7f64f97fa640 10 client.?.objecter ms_handle_connect 0x7f64fc0a98f0 2024-03-25T18:32:44.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.848+0000 7f64f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.852+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550482812 0 0) 0x7f64e0004eb0 con 0x7f64fc1430c0 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.853+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64fc137c90 con 0x7f64fc1430c0 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.853+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f64f40019b0 con 0x7f64fc1430c0 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.853+0000 7f64f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.853+0000 7f64f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.853+0000 7f64f97fa640 10 monclient(hunting): dump: 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:44.854 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.855 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:44.855 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:44.855 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:44.855 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:44.855 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:44.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.853+0000 7f64f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:44.860 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2324860156 0 0) 0x7f64fc137c90 con 0x7f64fc1430c0 2024-03-25T18:32:44.860 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 >> v1:172.21.15.102:6790/0 conn(0x7f64fc13f710 legacy=0x7f64fc1383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 >> v1:172.21.15.102:6789/0 conn(0x7f64fc0a98f0 legacy=0x7f64fc1db170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 1 monclient: found mon.b 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc138ad0 con 0x7f64fc1430c0 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 10 monclient: _finish_auth 0 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f64f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 10 monclient: _renew_subs 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64fc137880 con 0x7f64fc1430c0 2024-03-25T18:32:44.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:44.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 10 monclient: _renew_subs 2024-03-25T18:32:44.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:44.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc1e3520 con 0x7f64fc1430c0 2024-03-25T18:32:44.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.859+0000 7f6503065640 1 librados: init done 2024-03-25T18:32:44.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.860+0000 7f6503065640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:44.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.860+0000 7f6503065640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:44.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.860+0000 7f6503065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:44.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.860+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc0627b0 con 0x7f64fc1430c0 2024-03-25T18:32:44.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.864+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64f4001590 con 0x7f64fc1430c0 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.864+0000 7f64f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.864+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f64f4004ff0 con 0x7f64fc1430c0 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.864+0000 7f64f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.864+0000 7f64f97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.864+0000 7f64f97fa640 10 monclient: dump: 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:44.865 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:44.866 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:44.866 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:44.864+0000 7f64faffd640 4 set_mon_vals no callback set 2024-03-25T18:32:45.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.132+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f64f401afc0 con 0x7f64fc1430c0 2024-03-25T18:32:45.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.133+0000 7f6500e01640 10 monclient: get_auth_request con 0x7f64e006dd10 auth_method 0 2024-03-25T18:32:45.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.133+0000 7f64f97fa640 10 client.4540.objecter ms_handle_connect 0x7f64e006dd10 2024-03-25T18:32:45.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.136+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f64f4054c20 con 0x7f64fc1430c0 2024-03-25T18:32:45.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.136+0000 7f64f97fa640 10 client.4540.objecter ms_dispatch 0x7f64fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:45.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.136+0000 7f64f97fa640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:45.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.136+0000 7f64f97fa640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:45.137 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.136+0000 7f64f97fa640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T18:32:45.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.139+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64f4050560 con 0x7f64fc1430c0 2024-03-25T18:32:45.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.139+0000 7f64f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:45.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.139+0000 7f64f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:45.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.366+0000 7f6503065640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:45.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.366+0000 7f6503065640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:45.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.366+0000 7f6503065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:32:45.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.366+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64fc069f40 con 0x7f64fc1430c0 2024-03-25T18:32:45.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.389+0000 7f64f97fa640 1 -- 172.21.15.102:0/3078769570 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f64f4052660 con 0x7f64fc1430c0 2024-03-25T18:32:45.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.389+0000 7f64f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:45.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.389+0000 7f64f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:45.390 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:32:45.390 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:45.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.391+0000 7f6503065640 10 librados: watch_flush enter 2024-03-25T18:32:45.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 10 librados: watch_flush exit 2024-03-25T18:32:45.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T18:32:45.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 10 client.4540.objecter successfully canceled tick 2024-03-25T18:32:45.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 >> v1:172.21.15.146:6800/3848252714 conn(0x7f64e006dd10 legacy=0x7f64e00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:45.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 10 monclient: shutdown 2024-03-25T18:32:45.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:45.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 >> v1:172.21.15.146:6789/0 conn(0x7f64fc1430c0 legacy=0x7f64fc1e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:45.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 shutdown_connections 2024-03-25T18:32:45.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 >> 172.21.15.102:0/3078769570 conn(0x7f64fc0b7ea0 msgr2=0x7f64fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:45.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 shutdown_connections 2024-03-25T18:32:45.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 1 -- 172.21.15.102:0/3078769570 wait complete. 2024-03-25T18:32:45.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:45.392+0000 7f6503065640 1 librados: shutdown 2024-03-25T18:32:45.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:52.210 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:52.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.308+0000 7efcc3923640 10 monclient: get_monmap_and_config 2024-03-25T18:32:52.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.308+0000 7efcc3923640 10 monclient: build_initial_monmap 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.308+0000 7efcc3923640 10 monclient: monmap: 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:32:52.309423+0000 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:32:52.309423+0000 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:32:52.310 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.309+0000 7efcc3923640 1 Processor -- start 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.309+0000 7efcc3923640 1 -- start start 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.309+0000 7efcc3923640 10 monclient: init 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc0b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 10 monclient(hunting): picked mon.noname-b con 0x7efcbc0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc14be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0af940 con 0x7efcbc14be90 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0b0a70 con 0x7efcbc0b4d60 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc0b0d50 con 0x7efcbc0b1730 2024-03-25T18:32:52.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:52.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.310+0000 7efcc3923640 10 monclient(hunting): authenticate will time out at 7803.598282s 2024-03-25T18:32:52.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.319+0000 7efcc0ebe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7efcbc0b1730 0x7efcbc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:33968/0 (socket says 172.21.15.102:33968) 2024-03-25T18:32:52.320 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.319+0000 7efcc0ebe640 1 -- 172.21.15.102:0/3255692336 learned_addr learned my addr 172.21.15.102:0/3255692336 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:52.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.339+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912344622 0 0) 0x7efcbc0b0a70 con 0x7efcbc0b4d60 2024-03-25T18:32:52.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.339+0000 7efcb3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:52.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.339+0000 7efcb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:52.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.339+0000 7efcb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:52.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.339+0000 7efcb3fff640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:32:52.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.339+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca8003520 con 0x7efcbc0b4d60 2024-03-25T18:32:52.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.344+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684486326 0 0) 0x7efcbc0b0d50 con 0x7efcbc0b1730 2024-03-25T18:32:52.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.344+0000 7efcb3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:52.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.344+0000 7efcb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:52.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.344+0000 7efcb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:52.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.344+0000 7efcb3fff640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:32:52.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.344+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcbc0b0a70 con 0x7efcbc0b1730 2024-03-25T18:32:52.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.350+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068129205 0 0) 0x7efcbc0af940 con 0x7efcbc14be90 2024-03-25T18:32:52.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.350+0000 7efcb3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:32:52.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.350+0000 7efcb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:52.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.350+0000 7efcb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:52.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.350+0000 7efcb3fff640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:32:52.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.350+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcbc0b0d50 con 0x7efcbc14be90 2024-03-25T18:32:52.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2436876123 0 0) 0x7efcbc0b0a70 con 0x7efcbc0b1730 2024-03-25T18:32:52.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 >> v1:172.21.15.146:6789/0 conn(0x7efcbc0b4d60 legacy=0x7efcbc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 >> v1:172.21.15.102:6789/0 conn(0x7efcbc14be90 legacy=0x7efcbc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:32:52.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 1 monclient: found mon.noname-b 2024-03-25T18:32:52.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:52.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc0b1030 con 0x7efcbc0b1730 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 10 monclient: _finish_auth 0 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efcac001670 con 0x7efcbc0b1730 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcb3fff640 10 monclient: dump: 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:52.369 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:52.370 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:52.370 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:52.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcc3923640 5 monclient: authenticate success, global_id 4517 2024-03-25T18:32:52.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.367+0000 7efcc3923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:32:52.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.385+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcac0017d0 con 0x7efcbc0b1730 2024-03-25T18:32:52.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.385+0000 7efcb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:52.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.385+0000 7efcb3fff640 1 -- 172.21.15.102:0/3255692336 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efcac001be0 con 0x7efcbc0b1730 2024-03-25T18:32:52.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.385+0000 7efcc3923640 10 monclient: get_monmap_and_config success 2024-03-25T18:32:52.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.385+0000 7efcc3923640 4 set_mon_vals no callback set 2024-03-25T18:32:52.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.385+0000 7efcb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.385+0000 7efcb3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.385+0000 7efcb3fff640 10 monclient: dump: 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:52.387 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.386+0000 7efcc3923640 10 monclient: shutdown 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.386+0000 7efcc3923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.386+0000 7efcc3923640 1 -- 172.21.15.102:0/3255692336 >> v1:172.21.15.102:6790/0 conn(0x7efcbc0b1730 legacy=0x7efcbc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.386+0000 7efcc3923640 1 -- 172.21.15.102:0/3255692336 shutdown_connections 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.386+0000 7efcc3923640 1 -- 172.21.15.102:0/3255692336 >> 172.21.15.102:0/3255692336 conn(0x7efcbc0617a0 msgr2=0x7efcbc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.386+0000 7efcc3923640 1 -- 172.21.15.102:0/3255692336 shutdown_connections 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.386+0000 7efcc3923640 1 -- 172.21.15.102:0/3255692336 wait complete. 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 10 monclient: build_initial_monmap 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 10 monclient: monmap: 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:32:52.390 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 1 librados: starting msgr at 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 1 librados: starting objecter 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 1 Processor -- start 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 1 -- start start 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 1 librados: setting wanted keys 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 1 librados: calling monclient init 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.387+0000 7efcc3923640 10 monclient: init 2024-03-25T18:32:52.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 10 monclient(hunting): picked mon.noname-b con 0x7efcbc0b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc14be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc137b20 con 0x7efcbc14be90 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc135ba0 con 0x7efcbc0b4d60 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc136300 con 0x7efcbc0b1730 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:52.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.388+0000 7efcc3923640 10 monclient(hunting): authenticate will time out at 7803.676565s 2024-03-25T18:32:52.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.394+0000 7efcc1ec0640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efcbc14be90 0x7efcbc1db230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38042/0 (socket says 172.21.15.102:38042) 2024-03-25T18:32:52.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.394+0000 7efcc1ec0640 1 -- 172.21.15.102:0/428469874 learned_addr learned my addr 172.21.15.102:0/428469874 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:32:52.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.396+0000 7efcb1ffb640 10 client.?.objecter ms_handle_connect 0x7efcbc14be90 2024-03-25T18:32:52.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.396+0000 7efcb1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:52.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.400+0000 7efcb1ffb640 10 client.?.objecter ms_handle_connect 0x7efcbc0b4d60 2024-03-25T18:32:52.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.400+0000 7efcb1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:52.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.402+0000 7efcb1ffb640 10 client.?.objecter ms_handle_connect 0x7efcbc0b1730 2024-03-25T18:32:52.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.402+0000 7efcb1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:32:52.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.407+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981815587 0 0) 0x7efcbc137b20 con 0x7efcbc14be90 2024-03-25T18:32:52.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.407+0000 7efcb1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:52.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.407+0000 7efcb1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:52.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.407+0000 7efcb1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:52.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.407+0000 7efcb1ffb640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:32:52.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.407+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0004eb0 con 0x7efcbc14be90 2024-03-25T18:32:52.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.422+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124527246 0 0) 0x7efcbc135ba0 con 0x7efcbc0b4d60 2024-03-25T18:32:52.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.422+0000 7efcb1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:52.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.422+0000 7efcb1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:52.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.422+0000 7efcb1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:52.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.422+0000 7efcb1ffb640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:32:52.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.422+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcbc137b20 con 0x7efcbc0b4d60 2024-03-25T18:32:52.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.425+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042457385 0 0) 0x7efcbc136300 con 0x7efcbc0b1730 2024-03-25T18:32:52.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.425+0000 7efcb1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:32:52.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.425+0000 7efcb1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:32:52.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.425+0000 7efcb1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:32:52.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.425+0000 7efcb1ffb640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:32:52.426 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.425+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcbc135ba0 con 0x7efcbc0b1730 2024-03-25T18:32:52.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.428+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 264738987 0 0) 0x7efca0004eb0 con 0x7efcbc14be90 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.428+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcbc136300 con 0x7efcbc14be90 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.428+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efcb4001c70 con 0x7efcbc14be90 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.428+0000 7efcb1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.428+0000 7efcb1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.428+0000 7efcb1ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:52.430 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:52.431 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:52.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.429+0000 7efcb1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:32:52.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.434+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1119615642 0 0) 0x7efcbc137b20 con 0x7efcbc0b4d60 2024-03-25T18:32:52.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.434+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efca0004eb0 con 0x7efcbc0b4d60 2024-03-25T18:32:52.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.435+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efcac001ae0 con 0x7efcbc0b4d60 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.435+0000 7efcb1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.435+0000 7efcb1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.435+0000 7efcb1ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:52.436 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:52.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.444+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2742161360 0 0) 0x7efcbc135ba0 con 0x7efcbc0b1730 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.444+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcbc137b20 con 0x7efcbc0b1730 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.444+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efcb80017b0 con 0x7efcbc0b1730 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.444+0000 7efcb1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.444+0000 7efcb1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.444+0000 7efcb1ffb640 10 monclient(hunting): dump: 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:52.445 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:52.446 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:52.446 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 526920777 0 0) 0x7efcbc136300 con 0x7efcbc14be90 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 >> v1:172.21.15.102:6790/0 conn(0x7efcbc0b1730 legacy=0x7efcbc12fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 >> v1:172.21.15.146:6789/0 conn(0x7efcbc0b4d60 legacy=0x7efcbc1dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 1 monclient: found mon.a 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc136b70 con 0x7efcbc14be90 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcc3923640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:32:52.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcc3923640 10 monclient: _renew_subs 2024-03-25T18:32:52.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcc3923640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:52.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.448+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc145720 con 0x7efcbc14be90 2024-03-25T18:32:52.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.449+0000 7efcc3923640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:32:52.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.449+0000 7efcc3923640 10 monclient: _renew_subs 2024-03-25T18:32:52.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.449+0000 7efcc3923640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:52.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.449+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc130750 con 0x7efcbc14be90 2024-03-25T18:32:52.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.449+0000 7efcc3923640 1 librados: init done 2024-03-25T18:32:52.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.450+0000 7efcc3923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:52.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.450+0000 7efcc3923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:52.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.450+0000 7efcc3923640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:52.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.450+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc135d30 con 0x7efcbc14be90 2024-03-25T18:32:52.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.754+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcb4003990 con 0x7efcbc14be90 2024-03-25T18:32:52.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.754+0000 7efcb1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:32:52.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.754+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7efcb4003da0 con 0x7efcbc14be90 2024-03-25T18:32:52.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.754+0000 7efcb1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:32:52.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.754+0000 7efcb1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:32:52.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.754+0000 7efcb1ffb640 10 monclient: dump: 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.754+0000 7efcb37fe640 4 set_mon_vals no callback set 2024-03-25T18:32:52.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.754+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7efcb4003fc0 con 0x7efcbc14be90 2024-03-25T18:32:52.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.755+0000 7efcc16bf640 10 monclient: get_auth_request con 0x7efca006e0f0 auth_method 0 2024-03-25T18:32:52.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.755+0000 7efcb1ffb640 10 client.4545.objecter ms_handle_connect 0x7efca006e0f0 2024-03-25T18:32:52.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.761+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7efcb4054a10 con 0x7efcbc14be90 2024-03-25T18:32:52.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.761+0000 7efcb1ffb640 10 client.4545.objecter ms_dispatch 0x7efcbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:32:52.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.761+0000 7efcb1ffb640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:32:52.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.761+0000 7efcb1ffb640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:32:52.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.761+0000 7efcb1ffb640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T18:32:52.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.761+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efcb4050520 con 0x7efcbc14be90 2024-03-25T18:32:52.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.761+0000 7efcb1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:32:52.763 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:52.761+0000 7efcb1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:32:53.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.017+0000 7efcc3923640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:53.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.017+0000 7efcc3923640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:53.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.017+0000 7efcc3923640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:32:53.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.017+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcbc0627b0 con 0x7efcbc14be90 2024-03-25T18:32:53.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.048+0000 7efcb1ffb640 1 -- 172.21.15.102:0/428469874 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7efcb4052450 con 0x7efcbc14be90 2024-03-25T18:32:53.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.048+0000 7efcb1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:32:53.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.048+0000 7efcb1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:32:53.049 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:32:53.049 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:32:53.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.050+0000 7efcc3923640 10 librados: watch_flush enter 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.050+0000 7efcc3923640 10 librados: watch_flush exit 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.050+0000 7efcc3923640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.050+0000 7efcc3923640 10 client.4545.objecter successfully canceled tick 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.050+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 >> v1:172.21.15.146:6800/3848252714 conn(0x7efca006e0f0 legacy=0x7efca00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.050+0000 7efcc3923640 10 monclient: shutdown 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.050+0000 7efcc3923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.050+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 >> v1:172.21.15.102:6789/0 conn(0x7efcbc14be90 legacy=0x7efcbc1db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.051+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 shutdown_connections 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.051+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 >> 172.21.15.102:0/428469874 conn(0x7efcbc0617a0 msgr2=0x7efcbc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.051+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 shutdown_connections 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.051+0000 7efcc3923640 1 -- 172.21.15.102:0/428469874 wait complete. 2024-03-25T18:32:53.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:53.051+0000 7efcc3923640 1 librados: shutdown 2024-03-25T18:32:53.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:32:59.871 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:32:59.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.986+0000 7f03323fb640 10 monclient: get_monmap_and_config 2024-03-25T18:32:59.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.986+0000 7f03323fb640 10 monclient: build_initial_monmap 2024-03-25T18:32:59.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.987+0000 7f03323fb640 10 monclient: monmap: 2024-03-25T18:32:59.987 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:32:59.987643+0000 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:32:59.987643+0000 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:32:59.988 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:32:59.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.987+0000 7f03323fb640 1 Processor -- start 2024-03-25T18:32:59.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.987+0000 7f03323fb640 1 -- start start 2024-03-25T18:32:59.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.987+0000 7f03323fb640 10 monclient: init 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 10 monclient(hunting): picked mon.noname-a con 0x7f032c0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 10 monclient(hunting): picked mon.noname-c con 0x7f032c13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 10 monclient(hunting): picked mon.noname-b con 0x7f032c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c142520 con 0x7f032c0a98f0 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c0b28a0 con 0x7f032c13f710 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c0b2a30 con 0x7f032c1430c0 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:32:59.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.988+0000 7f03323fb640 10 monclient(hunting): authenticate will time out at 7811.276635s 2024-03-25T18:32:59.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.997+0000 7f032bfff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f032c0a98f0 0x7f032c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42846/0 (socket says 172.21.15.102:42846) 2024-03-25T18:32:59.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:32:59.997+0000 7f032bfff640 1 -- 172.21.15.102:0/4160644880 learned_addr learned my addr 172.21.15.102:0/4160644880 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:00.007 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.007+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106112732 0 0) 0x7f032c142520 con 0x7f032c0a98f0 2024-03-25T18:33:00.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.007+0000 7f032affd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:00.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.007+0000 7f032affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:00.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.007+0000 7f032affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:00.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.007+0000 7f032affd640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:33:00.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.007+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0314003520 con 0x7f032c0a98f0 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3940044493 0 0) 0x7f0314003520 con 0x7f032c0a98f0 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 >> v1:172.21.15.102:6790/0 conn(0x7f032c1430c0 legacy=0x7f032c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 >> v1:172.21.15.146:6789/0 conn(0x7f032c13f710 legacy=0x7f032c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 1 monclient: found mon.noname-a 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c0b44c0 con 0x7f032c0a98f0 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 10 monclient: _finish_auth 0 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f031c001690 con 0x7f032c0a98f0 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.015+0000 7f032affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:00.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.016+0000 7f032affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.016+0000 7f032affd640 10 monclient: dump: 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:00.018 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:00.019 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:00.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.016+0000 7f03323fb640 5 monclient: authenticate success, global_id 4548 2024-03-25T18:33:00.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.016+0000 7f03323fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:00.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.046+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f031c001860 con 0x7f032c0a98f0 2024-03-25T18:33:00.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.046+0000 7f032affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:00.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.046+0000 7f032affd640 1 -- 172.21.15.102:0/4160644880 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f031c001cb0 con 0x7f032c0a98f0 2024-03-25T18:33:00.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.046+0000 7f03323fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:00.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.046+0000 7f03323fb640 4 set_mon_vals no callback set 2024-03-25T18:33:00.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.046+0000 7f032affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:00.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.046+0000 7f032affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:00.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.046+0000 7f032affd640 10 monclient: dump: 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:00.048 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:00.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.047+0000 7f03323fb640 10 monclient: shutdown 2024-03-25T18:33:00.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.047+0000 7f03323fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.047+0000 7f03323fb640 1 -- 172.21.15.102:0/4160644880 >> v1:172.21.15.102:6789/0 conn(0x7f032c0a98f0 legacy=0x7f032c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.047+0000 7f03323fb640 1 -- 172.21.15.102:0/4160644880 shutdown_connections 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.047+0000 7f03323fb640 1 -- 172.21.15.102:0/4160644880 >> 172.21.15.102:0/4160644880 conn(0x7f032c0b7ea0 msgr2=0x7f032c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.047+0000 7f03323fb640 1 -- 172.21.15.102:0/4160644880 shutdown_connections 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.047+0000 7f03323fb640 1 -- 172.21.15.102:0/4160644880 wait complete. 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.048+0000 7f03323fb640 10 monclient: build_initial_monmap 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.048+0000 7f03323fb640 10 monclient: monmap: 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:33:00.050 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:33:00.051 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:00.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.048+0000 7f03323fb640 1 librados: starting msgr at 2024-03-25T18:33:00.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.048+0000 7f03323fb640 1 librados: starting objecter 2024-03-25T18:33:00.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.049+0000 7f03323fb640 1 Processor -- start 2024-03-25T18:33:00.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.049+0000 7f03323fb640 1 -- start start 2024-03-25T18:33:00.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.049+0000 7f03323fb640 1 librados: setting wanted keys 2024-03-25T18:33:00.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.049+0000 7f03323fb640 1 librados: calling monclient init 2024-03-25T18:33:00.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.049+0000 7f03323fb640 10 monclient: init 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 10 monclient(hunting): picked mon.noname-b con 0x7f032c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 10 monclient(hunting): picked mon.noname-a con 0x7f032c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 10 monclient(hunting): picked mon.noname-c con 0x7f032c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c1d85f0 con 0x7f032c13f710 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c1d76e0 con 0x7f032c0a98f0 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c137970 con 0x7f032c1430c0 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:00.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.050+0000 7f03323fb640 10 monclient(hunting): authenticate will time out at 7811.338249s 2024-03-25T18:33:00.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.056+0000 7f032b7fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f032c13f710 0x7f032c1383e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42876/0 (socket says 172.21.15.102:42876) 2024-03-25T18:33:00.057 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.056+0000 7f032b7fe640 1 -- 172.21.15.102:0/3672726841 learned_addr learned my addr 172.21.15.102:0/3672726841 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:00.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.061+0000 7f0328ff9640 10 client.?.objecter ms_handle_connect 0x7f032c13f710 2024-03-25T18:33:00.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.061+0000 7f0328ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:00.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.064+0000 7f0328ff9640 10 client.?.objecter ms_handle_connect 0x7f032c1430c0 2024-03-25T18:33:00.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.064+0000 7f0328ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:00.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.065+0000 7f0328ff9640 10 client.?.objecter ms_handle_connect 0x7f032c0a98f0 2024-03-25T18:33:00.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.065+0000 7f0328ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:00.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.067+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242868168 0 0) 0x7f032c137970 con 0x7f032c1430c0 2024-03-25T18:33:00.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.067+0000 7f0328ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:00.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.067+0000 7f0328ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:00.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.067+0000 7f0328ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:00.068 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.067+0000 7f0328ff9640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:33:00.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.067+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0318004eb0 con 0x7f032c1430c0 2024-03-25T18:33:00.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.075+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026015768 0 0) 0x7f032c1d85f0 con 0x7f032c13f710 2024-03-25T18:33:00.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.075+0000 7f0328ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:00.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.075+0000 7f0328ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:00.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.075+0000 7f0328ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:00.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.075+0000 7f0328ff9640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:33:00.076 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.075+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c137970 con 0x7f032c13f710 2024-03-25T18:33:00.079 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.079+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105603155 0 0) 0x7f032c1d76e0 con 0x7f032c0a98f0 2024-03-25T18:33:00.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.079+0000 7f0328ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:00.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.079+0000 7f0328ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:00.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.079+0000 7f0328ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:00.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.079+0000 7f0328ff9640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:33:00.080 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.079+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032c1d85f0 con 0x7f032c0a98f0 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.082+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3031328755 0 0) 0x7f0318004eb0 con 0x7f032c1430c0 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.082+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f032c1d76e0 con 0x7f032c1430c0 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.082+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0324001ad0 con 0x7f032c1430c0 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.082+0000 7f0328ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.082+0000 7f0328ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.082+0000 7f0328ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:00.083 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:00.084 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:00.084 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:00.084 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:00.084 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:00.084 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:00.084 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:00.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.082+0000 7f0328ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:00.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 65715618 0 0) 0x7f032c1d76e0 con 0x7f032c1430c0 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 >> v1:172.21.15.146:6789/0 conn(0x7f032c0a98f0 legacy=0x7f032c1d7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 >> v1:172.21.15.102:6789/0 conn(0x7f032c13f710 legacy=0x7f032c1383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 1 monclient: found mon.c 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c1d51f0 con 0x7f032c1430c0 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f0328ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 5 monclient: authenticate success, global_id 4523 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 10 monclient: _renew_subs 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:00.098 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c138af0 con 0x7f032c1430c0 2024-03-25T18:33:00.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:00.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 10 monclient: _renew_subs 2024-03-25T18:33:00.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:00.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c138e40 con 0x7f032c1430c0 2024-03-25T18:33:00.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.096+0000 7f03323fb640 1 librados: init done 2024-03-25T18:33:00.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.098+0000 7f03323fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:00.099 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.098+0000 7f03323fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:00.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.098+0000 7f03323fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:00.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.098+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032c1d7d30 con 0x7f032c1430c0 2024-03-25T18:33:00.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.404+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0324001fe0 con 0x7f032c1430c0 2024-03-25T18:33:00.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.404+0000 7f0328ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:00.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.404+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f0324003b10 con 0x7f032c1430c0 2024-03-25T18:33:00.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.404+0000 7f0328ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:00.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.404+0000 7f0328ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:00.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.404+0000 7f0328ff9640 10 monclient: dump: 2024-03-25T18:33:00.405 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.404+0000 7f032a7fc640 4 set_mon_vals no callback set 2024-03-25T18:33:00.406 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.404+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f0324003d30 con 0x7f032c1430c0 2024-03-25T18:33:00.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.405+0000 7f032bfff640 10 monclient: get_auth_request con 0x7f031806dda0 auth_method 0 2024-03-25T18:33:00.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.405+0000 7f0328ff9640 10 client.4523.objecter ms_handle_connect 0x7f031806dda0 2024-03-25T18:33:00.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.410+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f0324053820 con 0x7f032c1430c0 2024-03-25T18:33:00.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.410+0000 7f0328ff9640 10 client.4523.objecter ms_dispatch 0x7f032c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:00.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.410+0000 7f0328ff9640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:00.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.410+0000 7f0328ff9640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:00.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.410+0000 7f0328ff9640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T18:33:00.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.410+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f032404f330 con 0x7f032c1430c0 2024-03-25T18:33:00.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.410+0000 7f0328ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:00.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.410+0000 7f0328ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:00.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.628+0000 7f03323fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.628+0000 7f03323fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.628+0000 7f03323fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:00.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.628+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f032c0abb90 con 0x7f032c1430c0 2024-03-25T18:33:00.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.653+0000 7f0328ff9640 1 -- 172.21.15.102:0/3672726841 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f032c0abb90 con 0x7f032c1430c0 2024-03-25T18:33:00.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.653+0000 7f0328ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:00.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.653+0000 7f0328ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:00.655 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:33:00.655 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:00.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 10 librados: watch_flush enter 2024-03-25T18:33:00.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 10 librados: watch_flush exit 2024-03-25T18:33:00.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T18:33:00.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 10 client.4523.objecter successfully canceled tick 2024-03-25T18:33:00.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 >> v1:172.21.15.146:6800/3848252714 conn(0x7f031806dda0 legacy=0x7f0318070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 10 monclient: shutdown 2024-03-25T18:33:00.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:00.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 >> v1:172.21.15.102:6790/0 conn(0x7f032c1430c0 legacy=0x7f032c1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:00.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 shutdown_connections 2024-03-25T18:33:00.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 >> 172.21.15.102:0/3672726841 conn(0x7f032c0b7ea0 msgr2=0x7f032c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:00.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 shutdown_connections 2024-03-25T18:33:00.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 1 -- 172.21.15.102:0/3672726841 wait complete. 2024-03-25T18:33:00.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:00.655+0000 7f03323fb640 1 librados: shutdown 2024-03-25T18:33:00.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:04.112 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:20.347 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.122 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:20.357 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.156 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:20.392 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.169 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:20.382 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.179 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:20.392 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.182 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:20.417 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.190 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:20.425 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.205 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:20.418 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:04.213 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:20.449 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:07.474 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.588+0000 7f84b8b6d640 10 monclient: get_monmap_and_config 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.588+0000 7f84b8b6d640 10 monclient: build_initial_monmap 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.588+0000 7f84b8b6d640 10 monclient: monmap: 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:33:07.589904+0000 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:33:07.589904+0000 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:07.590 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:07.591 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:07.591 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:33:07.591 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:33:07.591 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.588+0000 7f84b8b6d640 1 Processor -- start 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.588+0000 7f84b8b6d640 1 -- start start 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.588+0000 7f84b8b6d640 10 monclient: init 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.589+0000 7f84b8b6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.589+0000 7f84b8b6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.589+0000 7f84b8b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f84b40a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.589+0000 7f84b8b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f84b413f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.589+0000 7f84b8b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f84b41430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.589+0000 7f84b8b6d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b4142520 con 0x7f84b41430c0 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.589+0000 7f84b8b6d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b40b28a0 con 0x7f84b40a98f0 2024-03-25T18:33:07.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.589+0000 7f84b8b6d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b40b2a30 con 0x7f84b413f710 2024-03-25T18:33:07.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.590+0000 7f84b8b6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:07.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.590+0000 7f84b8b6d640 10 monclient(hunting): authenticate will time out at 7818.878946s 2024-03-25T18:33:07.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.600+0000 7f84b2d9d640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f84b41430c0 0x7f84b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34936/0 (socket says 172.21.15.102:34936) 2024-03-25T18:33:07.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.600+0000 7f84b2d9d640 1 -- 172.21.15.102:0/1992875433 learned_addr learned my addr 172.21.15.102:0/1992875433 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:07.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.610+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104410996 0 0) 0x7f84b4142520 con 0x7f84b41430c0 2024-03-25T18:33:07.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.610+0000 7f84b159a640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.610+0000 7f84b159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.610+0000 7f84b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.610+0000 7f84b159a640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:33:07.612 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.610+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f849c003520 con 0x7f84b41430c0 2024-03-25T18:33:07.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.627+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143620120 0 0) 0x7f84b40b28a0 con 0x7f84b40a98f0 2024-03-25T18:33:07.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.627+0000 7f84b159a640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.627+0000 7f84b159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.627+0000 7f84b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.627+0000 7f84b159a640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:33:07.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.627+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b4142520 con 0x7f84b40a98f0 2024-03-25T18:33:07.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.628+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552839126 0 0) 0x7f84b40b2a30 con 0x7f84b413f710 2024-03-25T18:33:07.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.628+0000 7f84b159a640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.628+0000 7f84b159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.628+0000 7f84b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.628+0000 7f84b159a640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:33:07.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.628+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b40b28a0 con 0x7f84b413f710 2024-03-25T18:33:07.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1930847053 0 0) 0x7f84b4142520 con 0x7f84b40a98f0 2024-03-25T18:33:07.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 >> v1:172.21.15.102:6790/0 conn(0x7f84b413f710 legacy=0x7f84b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 >> v1:172.21.15.102:6789/0 conn(0x7f84b41430c0 legacy=0x7f84b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 10 monclient: _finish_hunting 0 2024-03-25T18:33:07.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 1 monclient: found mon.noname-c 2024-03-25T18:33:07.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b40b44c0 con 0x7f84b40a98f0 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 10 monclient: _finish_auth 0 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 10 monclient: _check_auth_tickets 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f84a4001670 con 0x7f84b40a98f0 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b159a640 10 monclient: dump: 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.638 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.639 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:07.639 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:07.639 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:07.639 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:07.639 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:07.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b8b6d640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:33:07.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.635+0000 7f84b8b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.658+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84a40017d0 con 0x7f84b40a98f0 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.658+0000 7f84b159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.658+0000 7f84b159a640 1 -- 172.21.15.102:0/1992875433 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f84a4001be0 con 0x7f84b40a98f0 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.658+0000 7f84b8b6d640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.658+0000 7f84b8b6d640 4 set_mon_vals no callback set 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.658+0000 7f84b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.658+0000 7f84b159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.658+0000 7f84b159a640 10 monclient: dump: 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:07.660 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.661 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.661 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.661 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:07.661 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:07.661 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:07.661 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:07.661 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.659+0000 7f84b8b6d640 10 monclient: shutdown 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.659+0000 7f84b8b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.659+0000 7f84b8b6d640 1 -- 172.21.15.102:0/1992875433 >> v1:172.21.15.146:6789/0 conn(0x7f84b40a98f0 legacy=0x7f84b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.659+0000 7f84b8b6d640 1 -- 172.21.15.102:0/1992875433 shutdown_connections 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.659+0000 7f84b8b6d640 1 -- 172.21.15.102:0/1992875433 >> 172.21.15.102:0/1992875433 conn(0x7f84b40b7ea0 msgr2=0x7f84b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.659+0000 7f84b8b6d640 1 -- 172.21.15.102:0/1992875433 shutdown_connections 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.659+0000 7f84b8b6d640 1 -- 172.21.15.102:0/1992875433 wait complete. 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 10 monclient: build_initial_monmap 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 10 monclient: monmap: 2024-03-25T18:33:07.663 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 1 librados: starting msgr at 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 1 librados: starting objecter 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 1 Processor -- start 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 1 -- start start 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 1 librados: setting wanted keys 2024-03-25T18:33:07.664 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 1 librados: calling monclient init 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.660+0000 7f84b8b6d640 10 monclient: init 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 10 monclient(hunting): picked mon.noname-b con 0x7f84b40a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 10 monclient(hunting): picked mon.noname-a con 0x7f84b413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 10 monclient(hunting): picked mon.noname-c con 0x7f84b41430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b41390f0 con 0x7f84b413f710 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b4138dd0 con 0x7f84b40a98f0 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b41346b0 con 0x7f84b41430c0 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:07.665 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.661+0000 7f84b8b6d640 10 monclient(hunting): authenticate will time out at 7818.950503s 2024-03-25T18:33:07.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.673+0000 7f84b2d9d640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f84b41430c0 0x7f84b4133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:33352/0 (socket says 172.21.15.102:33352) 2024-03-25T18:33:07.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.673+0000 7f84b2d9d640 1 -- 172.21.15.102:0/194300316 learned_addr learned my addr 172.21.15.102:0/194300316 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:07.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.674+0000 7f84a2ffd640 10 client.?.objecter ms_handle_connect 0x7f84b41430c0 2024-03-25T18:33:07.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.674+0000 7f84a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.676+0000 7f84a2ffd640 10 client.?.objecter ms_handle_connect 0x7f84b40a98f0 2024-03-25T18:33:07.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.676+0000 7f84a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.681+0000 7f84a2ffd640 10 client.?.objecter ms_handle_connect 0x7f84b413f710 2024-03-25T18:33:07.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.681+0000 7f84a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:07.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.684+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307275194 0 0) 0x7f84b4138dd0 con 0x7f84b40a98f0 2024-03-25T18:33:07.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.684+0000 7f84a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.684+0000 7f84a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.684+0000 7f84a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.684+0000 7f84a2ffd640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:33:07.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.684+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8494004eb0 con 0x7f84b40a98f0 2024-03-25T18:33:07.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.685+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197954783 0 0) 0x7f84b41346b0 con 0x7f84b41430c0 2024-03-25T18:33:07.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.685+0000 7f84a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.685+0000 7f84a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.685+0000 7f84a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.685+0000 7f84a2ffd640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:33:07.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.685+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b4138dd0 con 0x7f84b41430c0 2024-03-25T18:33:07.702 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.700+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862707156 0 0) 0x7f84b41390f0 con 0x7f84b413f710 2024-03-25T18:33:07.702 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.700+0000 7f84a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:07.702 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.700+0000 7f84a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:07.702 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.700+0000 7f84a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:07.702 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.700+0000 7f84a2ffd640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:33:07.702 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.700+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b41346b0 con 0x7f84b413f710 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.702+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 562009126 0 0) 0x7f84b4138dd0 con 0x7f84b41430c0 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.702+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84b41390f0 con 0x7f84b41430c0 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.702+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f84ac001b30 con 0x7f84b41430c0 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.702+0000 7f84a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.702+0000 7f84a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.702+0000 7f84a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.704 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.705 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:07.705 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:07.705 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:07.705 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:07.705 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:07.705 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.702+0000 7f84a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:07.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.705+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2654548004 0 0) 0x7f8494004eb0 con 0x7f84b40a98f0 2024-03-25T18:33:07.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.705+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84b4138dd0 con 0x7f84b40a98f0 2024-03-25T18:33:07.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.705+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f84a4001ba0 con 0x7f84b40a98f0 2024-03-25T18:33:07.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.705+0000 7f84a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.705+0000 7f84a2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:07.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.705+0000 7f84a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:07.707 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:07.708 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:07.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3620169843 0 0) 0x7f84b4138dd0 con 0x7f84b40a98f0 2024-03-25T18:33:07.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 >> v1:172.21.15.102:6790/0 conn(0x7f84b41430c0 legacy=0x7f84b4133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 >> v1:172.21.15.102:6789/0 conn(0x7f84b413f710 legacy=0x7f84b41386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:07.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:07.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 1 monclient: found mon.b 2024-03-25T18:33:07.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:07.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b4134840 con 0x7f84b40a98f0 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 5 monclient: authenticate success, global_id 4555 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 10 monclient: _renew_subs 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b4134a90 con 0x7f84b40a98f0 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 10 monclient: _renew_subs 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b41356f0 con 0x7f84b40a98f0 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.715+0000 7f84b8b6d640 1 librados: init done 2024-03-25T18:33:07.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.716+0000 7f84b8b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:07.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.716+0000 7f84b8b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:07.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.716+0000 7f84b8b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:07.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.716+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b40627b0 con 0x7f84b40a98f0 2024-03-25T18:33:07.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.996+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84a40038a0 con 0x7f84b40a98f0 2024-03-25T18:33:07.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.996+0000 7f84a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:07.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.996+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f84a4003bb0 con 0x7f84b40a98f0 2024-03-25T18:33:07.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.996+0000 7f84a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.996+0000 7f84a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.996+0000 7f84a2ffd640 10 monclient: dump: 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:07.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.996+0000 7f84b0d99640 4 set_mon_vals no callback set 2024-03-25T18:33:08.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.997+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f84a4019b80 con 0x7f84b40a98f0 2024-03-25T18:33:08.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.997+0000 7f84b1d9b640 10 monclient: get_auth_request con 0x7f849406df70 auth_method 0 2024-03-25T18:33:08.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:07.998+0000 7f84a2ffd640 10 client.4555.objecter ms_handle_connect 0x7f849406df70 2024-03-25T18:33:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.004+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f84a40536e0 con 0x7f84b40a98f0 2024-03-25T18:33:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.004+0000 7f84a2ffd640 10 client.4555.objecter ms_dispatch 0x7f84b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.004+0000 7f84a2ffd640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.004+0000 7f84a2ffd640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.004+0000 7f84a2ffd640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T18:33:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.005+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84a404f1f0 con 0x7f84b40a98f0 2024-03-25T18:33:08.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.005+0000 7f84a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:08.007 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.005+0000 7f84a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:08.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.221+0000 7f84b8b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:08.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.221+0000 7f84b8b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:08.224 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.221+0000 7f84b8b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:08.224 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.221+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84b413de90 con 0x7f84b40a98f0 2024-03-25T18:33:08.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.261+0000 7f84a2ffd640 1 -- 172.21.15.102:0/194300316 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f84a4051120 con 0x7f84b40a98f0 2024-03-25T18:33:08.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.261+0000 7f84a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:08.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.261+0000 7f84a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:08.263 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:33:08.263 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 10 librados: watch_flush enter 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 10 librados: watch_flush exit 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 10 client.4555.objecter successfully canceled tick 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 >> v1:172.21.15.146:6800/3848252714 conn(0x7f849406df70 legacy=0x7f8494070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 10 monclient: shutdown 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 >> v1:172.21.15.146:6789/0 conn(0x7f84b40a98f0 legacy=0x7f84b4137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 shutdown_connections 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.263+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 >> 172.21.15.102:0/194300316 conn(0x7f84b40b7ea0 msgr2=0x7f84b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:08.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.264+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 shutdown_connections 2024-03-25T18:33:08.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.264+0000 7f84b8b6d640 1 -- 172.21.15.102:0/194300316 wait complete. 2024-03-25T18:33:08.267 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:08.264+0000 7f84b8b6d640 1 librados: shutdown 2024-03-25T18:33:08.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:15.085 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:15.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.198+0000 7f9b783c1640 10 monclient: get_monmap_and_config 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.198+0000 7f9b783c1640 10 monclient: build_initial_monmap 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.198+0000 7f9b783c1640 10 monclient: monmap: 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:33:15.200275+0000 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:33:15.200275+0000 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:33:15.201 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:33:15.202 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:15.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.199+0000 7f9b783c1640 1 Processor -- start 2024-03-25T18:33:15.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.199+0000 7f9b783c1640 1 -- start start 2024-03-25T18:33:15.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.199+0000 7f9b783c1640 10 monclient: init 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 10 monclient(hunting): picked mon.noname-b con 0x7f9b700b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 10 monclient(hunting): picked mon.noname-a con 0x7f9b700b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7014be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b700af940 con 0x7f9b700b1730 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b700b0a70 con 0x7f9b7014be90 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b700b0d50 con 0x7f9b700b4d60 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:15.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.200+0000 7f9b783c1640 10 monclient(hunting): authenticate will time out at 7826.489564s 2024-03-25T18:33:15.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.210+0000 7f9b7695e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9b7014be90 0x7f9b7014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:60314/0 (socket says 172.21.15.102:60314) 2024-03-25T18:33:15.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.210+0000 7f9b7695e640 1 -- 172.21.15.102:0/754286295 learned_addr learned my addr 172.21.15.102:0/754286295 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:15.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.221+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489507646 0 0) 0x7f9b700af940 con 0x7f9b700b1730 2024-03-25T18:33:15.223 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.221+0000 7f9b7515b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.224 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.221+0000 7f9b7515b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.224 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.221+0000 7f9b7515b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.224 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.221+0000 7f9b7515b640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:33:15.224 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.222+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c003520 con 0x7f9b700b1730 2024-03-25T18:33:15.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.224+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502352480 0 0) 0x7f9b700b0d50 con 0x7f9b700b4d60 2024-03-25T18:33:15.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.224+0000 7f9b7515b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.225 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.224+0000 7f9b7515b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.224+0000 7f9b7515b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.224+0000 7f9b7515b640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:33:15.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.224+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b700af940 con 0x7f9b700b4d60 2024-03-25T18:33:15.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.225+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548693298 0 0) 0x7f9b700b0a70 con 0x7f9b7014be90 2024-03-25T18:33:15.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.225+0000 7f9b7515b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.225+0000 7f9b7515b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.225+0000 7f9b7515b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.225+0000 7f9b7515b640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:33:15.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.225+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b700b0d50 con 0x7f9b7014be90 2024-03-25T18:33:15.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3542476014 0 0) 0x7f9b5c003520 con 0x7f9b700b1730 2024-03-25T18:33:15.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 >> v1:172.21.15.102:6790/0 conn(0x7f9b700b4d60 legacy=0x7f9b7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 >> v1:172.21.15.146:6789/0 conn(0x7f9b7014be90 legacy=0x7f9b7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:15.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 1 monclient: found mon.noname-a 2024-03-25T18:33:15.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:15.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:33:15.233 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b700b1030 con 0x7f9b700b1730 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 10 monclient: _finish_auth 0 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b60001690 con 0x7f9b700b1730 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b783c1640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b783c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b7515b640 10 monclient: dump: 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:15.234 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.235 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.235 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.235 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:15.235 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:15.235 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:15.235 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:15.235 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:15.235 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.231+0000 7f9b783c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:15.247 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.245+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b600018b0 con 0x7f9b700b1730 2024-03-25T18:33:15.247 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.245+0000 7f9b7515b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:15.247 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.245+0000 7f9b7515b640 1 -- 172.21.15.102:0/754286295 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b60001c40 con 0x7f9b700b1730 2024-03-25T18:33:15.247 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.245+0000 7f9b7515b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.245+0000 7f9b7515b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.245+0000 7f9b7515b640 10 monclient: dump: 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.245+0000 7f9b783c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:15.248 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.245+0000 7f9b783c1640 4 set_mon_vals no callback set 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.246+0000 7f9b783c1640 10 monclient: shutdown 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.246+0000 7f9b783c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.246+0000 7f9b783c1640 1 -- 172.21.15.102:0/754286295 >> v1:172.21.15.102:6789/0 conn(0x7f9b700b1730 legacy=0x7f9b700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.246+0000 7f9b783c1640 1 -- 172.21.15.102:0/754286295 shutdown_connections 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.246+0000 7f9b783c1640 1 -- 172.21.15.102:0/754286295 >> 172.21.15.102:0/754286295 conn(0x7f9b700617a0 msgr2=0x7f9b70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.246+0000 7f9b783c1640 1 -- 172.21.15.102:0/754286295 shutdown_connections 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.246+0000 7f9b783c1640 1 -- 172.21.15.102:0/754286295 wait complete. 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.247+0000 7f9b783c1640 10 monclient: build_initial_monmap 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.247+0000 7f9b783c1640 10 monclient: monmap: 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:15.251 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.247+0000 7f9b783c1640 1 librados: starting msgr at 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.247+0000 7f9b783c1640 1 librados: starting objecter 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 1 Processor -- start 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 1 -- start start 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 1 librados: setting wanted keys 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 1 librados: calling monclient init 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 10 monclient: init 2024-03-25T18:33:15.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 10 monclient(hunting): picked mon.noname-c con 0x7f9b700b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 10 monclient(hunting): picked mon.noname-a con 0x7f9b700b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7014be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b701daec0 con 0x7f9b700b4d60 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b701dbd40 con 0x7f9b7014be90 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7013a510 con 0x7f9b700b1730 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:15.253 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.248+0000 7f9b783c1640 10 monclient(hunting): authenticate will time out at 7826.537660s 2024-03-25T18:33:15.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.253+0000 7f9b7695e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9b7014be90 0x7f9b701e3260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:60366/0 (socket says 172.21.15.102:60366) 2024-03-25T18:33:15.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.253+0000 7f9b7695e640 1 -- 172.21.15.102:0/1345813082 learned_addr learned my addr 172.21.15.102:0/1345813082 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:15.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.255+0000 7f9b66ffd640 10 client.?.objecter ms_handle_connect 0x7f9b7014be90 2024-03-25T18:33:15.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.255+0000 7f9b66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:15.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.267+0000 7f9b66ffd640 10 client.?.objecter ms_handle_connect 0x7f9b700b1730 2024-03-25T18:33:15.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.267+0000 7f9b66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:15.273 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.271+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471811062 0 0) 0x7f9b701dbd40 con 0x7f9b7014be90 2024-03-25T18:33:15.273 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.271+0000 7f9b66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.273 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.271+0000 7f9b66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.273 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.271+0000 7f9b66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.271+0000 7f9b66ffd640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:33:15.274 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.271+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b54004eb0 con 0x7f9b7014be90 2024-03-25T18:33:15.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.274+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647950700 0 0) 0x7f9b7013a510 con 0x7f9b700b1730 2024-03-25T18:33:15.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.274+0000 7f9b66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.274+0000 7f9b66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.274+0000 7f9b66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.274+0000 7f9b66ffd640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:33:15.276 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.274+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b701dbd40 con 0x7f9b700b1730 2024-03-25T18:33:15.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.277+0000 7f9b66ffd640 10 client.?.objecter ms_handle_connect 0x7f9b700b4d60 2024-03-25T18:33:15.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.277+0000 7f9b66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:15.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.281+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2517391735 0 0) 0x7f9b701dbd40 con 0x7f9b700b1730 2024-03-25T18:33:15.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.281+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b7013a510 con 0x7f9b700b1730 2024-03-25T18:33:15.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.281+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b6c001970 con 0x7f9b700b1730 2024-03-25T18:33:15.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.281+0000 7f9b66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.281+0000 7f9b66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.281+0000 7f9b66ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:15.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.281+0000 7f9b66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:15.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.282+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 601588500 0 0) 0x7f9b54004eb0 con 0x7f9b7014be90 2024-03-25T18:33:15.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.282+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b701dbd40 con 0x7f9b7014be90 2024-03-25T18:33:15.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.282+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b68001bd0 con 0x7f9b7014be90 2024-03-25T18:33:15.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.282+0000 7f9b66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.282+0000 7f9b66ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:15.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.282+0000 7f9b66ffd640 10 monclient(hunting): dump: 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.283+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689984892 0 0) 0x7f9b701daec0 con 0x7f9b700b4d60 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.283+0000 7f9b66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.283+0000 7f9b66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:15.286 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.283+0000 7f9b66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:15.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.283+0000 7f9b66ffd640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:33:15.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.283+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b54004eb0 con 0x7f9b700b4d60 2024-03-25T18:33:15.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4226696380 0 0) 0x7f9b7013a510 con 0x7f9b700b1730 2024-03-25T18:33:15.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 >> v1:172.21.15.146:6789/0 conn(0x7f9b7014be90 legacy=0x7f9b701e3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 >> v1:172.21.15.102:6789/0 conn(0x7f9b700b4d60 legacy=0x7f9b7013b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:15.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 1 monclient: found mon.c 2024-03-25T18:33:15.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7013a7f0 con 0x7f9b700b1730 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 5 monclient: authenticate success, global_id 4535 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 10 monclient: _renew_subs 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b701db110 con 0x7f9b700b1730 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 10 monclient: _renew_subs 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7013c090 con 0x7f9b700b1730 2024-03-25T18:33:15.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.292+0000 7f9b783c1640 1 librados: init done 2024-03-25T18:33:15.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.293+0000 7f9b783c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.293+0000 7f9b783c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.293+0000 7f9b783c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:15.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.293+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b700627b0 con 0x7f9b700b1730 2024-03-25T18:33:15.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.581+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b6c003610 con 0x7f9b700b1730 2024-03-25T18:33:15.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.581+0000 7f9b66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:15.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.581+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f9b6c003920 con 0x7f9b700b1730 2024-03-25T18:33:15.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.581+0000 7f9b66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:15.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.581+0000 7f9b66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:15.583 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.581+0000 7f9b66ffd640 10 monclient: dump: 2024-03-25T18:33:15.583 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:15.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.581+0000 7f9b7495a640 4 set_mon_vals no callback set 2024-03-25T18:33:15.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.582+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f9b6c0198f0 con 0x7f9b700b1730 2024-03-25T18:33:15.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.582+0000 7f9b7595c640 10 monclient: get_auth_request con 0x7f9b5406deb0 auth_method 0 2024-03-25T18:33:15.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.583+0000 7f9b66ffd640 10 client.4535.objecter ms_handle_connect 0x7f9b5406deb0 2024-03-25T18:33:15.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.590+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f9b6c0535f0 con 0x7f9b700b1730 2024-03-25T18:33:15.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.590+0000 7f9b66ffd640 10 client.4535.objecter ms_dispatch 0x7f9b70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:15.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.590+0000 7f9b66ffd640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:15.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.590+0000 7f9b66ffd640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:15.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.590+0000 7f9b66ffd640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-25T18:33:15.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.590+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b6c04ef30 con 0x7f9b700b1730 2024-03-25T18:33:15.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.590+0000 7f9b66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:15.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.590+0000 7f9b66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:15.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.842+0000 7f9b783c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.842+0000 7f9b783c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.842+0000 7f9b783c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:15.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.842+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b70069f40 con 0x7f9b700b1730 2024-03-25T18:33:15.866 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.864+0000 7f9b66ffd640 1 -- 172.21.15.102:0/1345813082 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f9b6c051030 con 0x7f9b700b1730 2024-03-25T18:33:15.866 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.864+0000 7f9b66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:15.866 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.864+0000 7f9b66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:15.866 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:33:15.866 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 10 librados: watch_flush enter 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 10 librados: watch_flush exit 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 10 client.4535.objecter successfully canceled tick 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 >> v1:172.21.15.146:6800/3848252714 conn(0x7f9b5406deb0 legacy=0x7f9b54070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 10 monclient: shutdown 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 >> v1:172.21.15.102:6790/0 conn(0x7f9b700b1730 legacy=0x7f9b701db630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 shutdown_connections 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.866+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 >> 172.21.15.102:0/1345813082 conn(0x7f9b700617a0 msgr2=0x7f9b700b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.867+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 shutdown_connections 2024-03-25T18:33:15.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.867+0000 7f9b783c1640 1 -- 172.21.15.102:0/1345813082 wait complete. 2024-03-25T18:33:15.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:15.867+0000 7f9b783c1640 1 librados: shutdown 2024-03-25T18:33:15.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:22.688 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:22.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.804+0000 7f37a3e57640 10 monclient: get_monmap_and_config 2024-03-25T18:33:22.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.804+0000 7f37a3e57640 10 monclient: build_initial_monmap 2024-03-25T18:33:22.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.804+0000 7f37a3e57640 10 monclient: monmap: 2024-03-25T18:33:22.805 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:33:22.805179+0000 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:33:22.805179+0000 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:33:22.806 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:22.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.805+0000 7f37a3e57640 1 Processor -- start 2024-03-25T18:33:22.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.805+0000 7f37a3e57640 1 -- start start 2024-03-25T18:33:22.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.805+0000 7f37a3e57640 10 monclient: init 2024-03-25T18:33:22.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 10 monclient(hunting): picked mon.noname-a con 0x7f379c0b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 10 monclient(hunting): picked mon.noname-b con 0x7f379c0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 10 monclient(hunting): picked mon.noname-c con 0x7f379c14be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0af940 con 0x7f379c0b4d60 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0b0a70 con 0x7f379c14be90 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0b0d50 con 0x7f379c0b1730 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:22.808 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.806+0000 7f37a3e57640 10 monclient(hunting): authenticate will time out at 7834.094231s 2024-03-25T18:33:22.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.810+0000 7f37a23f4640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f379c14be90 0x7f379c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:54124/0 (socket says 172.21.15.102:54124) 2024-03-25T18:33:22.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.810+0000 7f37a23f4640 1 -- 172.21.15.102:0/2040982071 learned_addr learned my addr 172.21.15.102:0/2040982071 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:22.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.822+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723717067 0 0) 0x7f379c0af940 con 0x7f379c0b4d60 2024-03-25T18:33:22.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.822+0000 7f37a0bf1640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.822+0000 7f37a0bf1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.822+0000 7f37a0bf1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.822+0000 7f37a0bf1640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:33:22.823 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.822+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3788003520 con 0x7f379c0b4d60 2024-03-25T18:33:22.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.828+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472039276 0 0) 0x7f379c0b0a70 con 0x7f379c14be90 2024-03-25T18:33:22.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.828+0000 7f37a0bf1640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.828 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.828+0000 7f37a0bf1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.828+0000 7f37a0bf1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.828+0000 7f37a0bf1640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:33:22.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.828+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379c0af940 con 0x7f379c14be90 2024-03-25T18:33:22.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3222286351 0 0) 0x7f3788003520 con 0x7f379c0b4d60 2024-03-25T18:33:22.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 >> v1:172.21.15.102:6790/0 conn(0x7f379c0b1730 legacy=0x7f379c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 >> v1:172.21.15.146:6789/0 conn(0x7f379c14be90 legacy=0x7f379c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 10 monclient: _finish_hunting 0 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 1 monclient: found mon.noname-a 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c0b1030 con 0x7f379c0b4d60 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 10 monclient: _finish_auth 0 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 10 monclient: _check_auth_tickets 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a3e57640 5 monclient: authenticate success, global_id 4566 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3798001690 con 0x7f379c0b4d60 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a0bf1640 10 monclient: dump: 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:22.843 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:22.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.841+0000 7f37a3e57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:22.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.863+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37980017f0 con 0x7f379c0b4d60 2024-03-25T18:33:22.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.863+0000 7f37a0bf1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:22.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.863+0000 7f37a0bf1640 1 -- 172.21.15.102:0/2040982071 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3798001c00 con 0x7f379c0b4d60 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.863+0000 7f37a3e57640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.863+0000 7f37a3e57640 4 set_mon_vals no callback set 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.863+0000 7f37a0bf1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a0bf1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a0bf1640 10 monclient: dump: 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:22.865 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:22.866 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:22.866 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a3e57640 10 monclient: shutdown 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a3e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a3e57640 1 -- 172.21.15.102:0/2040982071 >> v1:172.21.15.102:6789/0 conn(0x7f379c0b4d60 legacy=0x7f379c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a3e57640 1 -- 172.21.15.102:0/2040982071 shutdown_connections 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a3e57640 1 -- 172.21.15.102:0/2040982071 >> 172.21.15.102:0/2040982071 conn(0x7f379c0617a0 msgr2=0x7f379c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a3e57640 1 -- 172.21.15.102:0/2040982071 shutdown_connections 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.864+0000 7f37a3e57640 1 -- 172.21.15.102:0/2040982071 wait complete. 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.865+0000 7f37a3e57640 10 monclient: build_initial_monmap 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.865+0000 7f37a3e57640 10 monclient: monmap: 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:22.868 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.865+0000 7f37a3e57640 1 librados: starting msgr at 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.865+0000 7f37a3e57640 1 librados: starting objecter 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 1 Processor -- start 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 1 -- start start 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 1 librados: setting wanted keys 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 1 librados: calling monclient init 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 10 monclient: init 2024-03-25T18:33:22.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 10 monclient(hunting): picked mon.noname-c con 0x7f379c0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 10 monclient(hunting): picked mon.noname-a con 0x7f379c0b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 10 monclient(hunting): picked mon.noname-b con 0x7f379c14be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c136130 con 0x7f379c0b4d60 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c145800 con 0x7f379c14be90 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c1452b0 con 0x7f379c0b1730 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:22.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.866+0000 7f37a3e57640 10 monclient(hunting): authenticate will time out at 7834.154915s 2024-03-25T18:33:22.881 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.880+0000 7f37a13f2640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f379c0b4d60 0x7f379c1db3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51486/0 (socket says 172.21.15.102:51486) 2024-03-25T18:33:22.881 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.880+0000 7f37a13f2640 1 -- 172.21.15.102:0/1251831050 learned_addr learned my addr 172.21.15.102:0/1251831050 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:22.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.881+0000 7f37927fc640 10 client.?.objecter ms_handle_connect 0x7f379c0b4d60 2024-03-25T18:33:22.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.881+0000 7f37927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.883 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.883+0000 7f37927fc640 10 client.?.objecter ms_handle_connect 0x7f379c0b1730 2024-03-25T18:33:22.884 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.883+0000 7f37927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.889 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.888+0000 7f37927fc640 10 client.?.objecter ms_handle_connect 0x7f379c14be90 2024-03-25T18:33:22.889 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.888+0000 7f37927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:22.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.894+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922201809 0 0) 0x7f379c136130 con 0x7f379c0b4d60 2024-03-25T18:33:22.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.894+0000 7f37927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.894+0000 7f37927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.894+0000 7f37927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.894+0000 7f37927fc640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:33:22.895 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.894+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780004eb0 con 0x7f379c0b4d60 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066747478 0 0) 0x7f379c1452b0 con 0x7f379c0b1730 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379c136130 con 0x7f379c0b1730 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141294094 0 0) 0x7f379c145800 con 0x7f379c14be90 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:22.905 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:33:22.906 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.904+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379c1452b0 con 0x7f379c14be90 2024-03-25T18:33:22.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.919+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668313026 0 0) 0x7f379c1452b0 con 0x7f379c14be90 2024-03-25T18:33:22.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.919+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f379c145800 con 0x7f379c14be90 2024-03-25T18:33:22.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.919+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3794001b90 con 0x7f379c14be90 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.919+0000 7f37927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.919+0000 7f37927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.919+0000 7f37927fc640 10 monclient(hunting): dump: 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:22.921 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.919+0000 7f37927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:22.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.921+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2724615782 0 0) 0x7f3780004eb0 con 0x7f379c0b4d60 2024-03-25T18:33:22.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.921+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f379c1452b0 con 0x7f379c0b4d60 2024-03-25T18:33:22.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.921+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f378c001a30 con 0x7f379c0b4d60 2024-03-25T18:33:22.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.921+0000 7f37927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:22.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.921+0000 7f37927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:22.922 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.921+0000 7f37927fc640 10 monclient(hunting): dump: 2024-03-25T18:33:22.922 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:22.923 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2550938518 0 0) 0x7f379c1452b0 con 0x7f379c0b4d60 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 >> v1:172.21.15.102:6790/0 conn(0x7f379c0b1730 legacy=0x7f379c1dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 >> v1:172.21.15.146:6789/0 conn(0x7f379c14be90 legacy=0x7f379c1e2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 1 monclient: found mon.a 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c145440 con 0x7f379c0b4d60 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:22.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37a3e57640 5 monclient: authenticate success, global_id 4569 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37a3e57640 10 monclient: _renew_subs 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37a3e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c135c10 con 0x7f379c0b4d60 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37a3e57640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37a3e57640 10 monclient: _renew_subs 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37a3e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.933+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c13e3f0 con 0x7f379c0b4d60 2024-03-25T18:33:22.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.934+0000 7f37a3e57640 1 librados: init done 2024-03-25T18:33:22.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.935+0000 7f37a3e57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:22.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.935+0000 7f37a3e57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:22.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.935+0000 7f37a3e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:22.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:22.935+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379c1362c0 con 0x7f379c0b4d60 2024-03-25T18:33:23.236 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.235+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f378c001c70 con 0x7f379c0b4d60 2024-03-25T18:33:23.236 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.235+0000 7f37927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:23.236 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.235+0000 7f3793fff640 4 set_mon_vals no callback set 2024-03-25T18:33:23.236 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.235+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f378c003a60 con 0x7f379c0b4d60 2024-03-25T18:33:23.236 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.235+0000 7f37927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:23.236 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.235+0000 7f37927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.235+0000 7f37927fc640 10 monclient: dump: 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:23.237 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:23.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.236+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f378c003c80 con 0x7f379c0b4d60 2024-03-25T18:33:23.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.236+0000 7f37a1bf3640 10 monclient: get_auth_request con 0x7f378006dfb0 auth_method 0 2024-03-25T18:33:23.238 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.237+0000 7f37927fc640 10 client.4569.objecter ms_handle_connect 0x7f378006dfb0 2024-03-25T18:33:23.242 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.241+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f378c054870 con 0x7f379c0b4d60 2024-03-25T18:33:23.242 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.241+0000 7f37927fc640 10 client.4569.objecter ms_dispatch 0x7f379c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:23.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.241+0000 7f37927fc640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:23.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.241+0000 7f37927fc640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:23.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.241+0000 7f37927fc640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-25T18:33:23.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.242+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f378c050380 con 0x7f379c0b4d60 2024-03-25T18:33:23.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.242+0000 7f37927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:23.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.242+0000 7f37927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:23.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.486+0000 7f37a3e57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.487 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.487+0000 7f37a3e57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.487+0000 7f37a3e57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:23.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.487+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f379c135f60 con 0x7f379c0b4d60 2024-03-25T18:33:23.517 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.517+0000 7f37927fc640 1 -- 172.21.15.102:0/1251831050 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f378c0522b0 con 0x7f379c0b4d60 2024-03-25T18:33:23.518 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.517+0000 7f37927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:23.518 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.517+0000 7f37927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:23.518 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:33:23.518 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:23.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 10 librados: watch_flush enter 2024-03-25T18:33:23.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 10 librados: watch_flush exit 2024-03-25T18:33:23.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-25T18:33:23.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 10 client.4569.objecter successfully canceled tick 2024-03-25T18:33:23.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 >> v1:172.21.15.146:6800/3848252714 conn(0x7f378006dfb0 legacy=0x7f3780070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:23.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 10 monclient: shutdown 2024-03-25T18:33:23.521 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:23.521 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 >> v1:172.21.15.102:6789/0 conn(0x7f379c0b4d60 legacy=0x7f379c1db3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:23.521 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 shutdown_connections 2024-03-25T18:33:23.521 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 >> 172.21.15.102:0/1251831050 conn(0x7f379c0617a0 msgr2=0x7f379c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:23.521 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 shutdown_connections 2024-03-25T18:33:23.521 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 1 -- 172.21.15.102:0/1251831050 wait complete. 2024-03-25T18:33:23.521 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:23.519+0000 7f37a3e57640 1 librados: shutdown 2024-03-25T18:33:23.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:24.164 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:40.400 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.172 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:40.408 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.197 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:07:40.433 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.222 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:40.458 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.226 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:40.462 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.228 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:40.441 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.239 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:40.451 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.264 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:07:40.500 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:24.283 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:07:40.496 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:30.340 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:30.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.444+0000 7f2b7a486640 10 monclient: get_monmap_and_config 2024-03-25T18:33:30.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.444+0000 7f2b7a486640 10 monclient: build_initial_monmap 2024-03-25T18:33:30.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.444+0000 7f2b7a486640 10 monclient: monmap: 2024-03-25T18:33:30.445 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:30.445 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:30.445 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:33:30.445246+0000 2024-03-25T18:33:30.445 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:33:30.445246+0000 2024-03-25T18:33:30.446 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:30.446 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:30.446 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:30.446 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:33:30.446 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:33:30.446 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:30.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.445+0000 7f2b7a486640 1 Processor -- start 2024-03-25T18:33:30.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.445+0000 7f2b7a486640 1 -- start start 2024-03-25T18:33:30.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.445+0000 7f2b7a486640 10 monclient: init 2024-03-25T18:33:30.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.445+0000 7f2b7a486640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:30.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.445+0000 7f2b7a486640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:30.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.445+0000 7f2b7a486640 10 monclient(hunting): picked mon.noname-c con 0x7f2b740a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:30.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.445+0000 7f2b7a486640 10 monclient(hunting): picked mon.noname-a con 0x7f2b7413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:30.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.446+0000 7f2b7a486640 10 monclient(hunting): picked mon.noname-b con 0x7f2b741430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:30.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.446+0000 7f2b7a486640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b74142520 con 0x7f2b7413f710 2024-03-25T18:33:30.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.446+0000 7f2b7a486640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740b28a0 con 0x7f2b740a98f0 2024-03-25T18:33:30.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.446+0000 7f2b7a486640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740b2a30 con 0x7f2b741430c0 2024-03-25T18:33:30.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.446+0000 7f2b7a486640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:30.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.446+0000 7f2b7a486640 10 monclient(hunting): authenticate will time out at 7841.733992s 2024-03-25T18:33:30.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.456+0000 7f2b78a23640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f2b741430c0 0x7f2b7414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:43164/0 (socket says 172.21.15.102:43164) 2024-03-25T18:33:30.457 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.456+0000 7f2b78a23640 1 -- 172.21.15.102:0/1162764483 learned_addr learned my addr 172.21.15.102:0/1162764483 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:30.483 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.483+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354585256 0 0) 0x7f2b740b2a30 con 0x7f2b741430c0 2024-03-25T18:33:30.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.483+0000 7f2b72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.483+0000 7f2b72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.483+0000 7f2b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.483+0000 7f2b72ffd640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:33:30.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.483+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b5c003520 con 0x7f2b741430c0 2024-03-25T18:33:30.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187121305 0 0) 0x7f2b5c003520 con 0x7f2b741430c0 2024-03-25T18:33:30.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 >> v1:172.21.15.146:6789/0 conn(0x7f2b740a98f0 legacy=0x7f2b740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 >> v1:172.21.15.102:6789/0 conn(0x7f2b7413f710 legacy=0x7f2b741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.500 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 1 monclient: found mon.noname-b 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b740b44c0 con 0x7f2b741430c0 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 10 monclient: _finish_auth 0 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2b6c001a10 con 0x7f2b741430c0 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b7a486640 5 monclient: authenticate success, global_id 4544 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b72ffd640 10 monclient: dump: 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:30.501 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:30.502 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.499+0000 7f2b7a486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:30.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.573+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b6c001b70 con 0x7f2b741430c0 2024-03-25T18:33:30.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.573+0000 7f2b72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:30.574 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.573+0000 7f2b72ffd640 1 -- 172.21.15.102:0/1162764483 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2b6c001f80 con 0x7f2b741430c0 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.573+0000 7f2b72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.573+0000 7f2b72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.573+0000 7f2b72ffd640 10 monclient: dump: 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.573+0000 7f2b7a486640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:30.575 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.573+0000 7f2b7a486640 4 set_mon_vals no callback set 2024-03-25T18:33:30.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.574+0000 7f2b7a486640 10 monclient: shutdown 2024-03-25T18:33:30.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.574+0000 7f2b7a486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.574+0000 7f2b7a486640 1 -- 172.21.15.102:0/1162764483 >> v1:172.21.15.102:6790/0 conn(0x7f2b741430c0 legacy=0x7f2b7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.574+0000 7f2b7a486640 1 -- 172.21.15.102:0/1162764483 shutdown_connections 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.574+0000 7f2b7a486640 1 -- 172.21.15.102:0/1162764483 >> 172.21.15.102:0/1162764483 conn(0x7f2b740b7ea0 msgr2=0x7f2b740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.574+0000 7f2b7a486640 1 -- 172.21.15.102:0/1162764483 shutdown_connections 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.574+0000 7f2b7a486640 1 -- 172.21.15.102:0/1162764483 wait complete. 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.575+0000 7f2b7a486640 10 monclient: build_initial_monmap 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.575+0000 7f2b7a486640 10 monclient: monmap: 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:33:30.578 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 1 librados: starting msgr at 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 1 librados: starting objecter 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 1 Processor -- start 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 1 -- start start 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 1 librados: setting wanted keys 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 1 librados: calling monclient init 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 10 monclient: init 2024-03-25T18:33:30.579 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 10 monclient(hunting): picked mon.noname-c con 0x7f2b740a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 10 monclient(hunting): picked mon.noname-a con 0x7f2b7413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.576+0000 7f2b7a486640 10 monclient(hunting): picked mon.noname-b con 0x7f2b741430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.577+0000 7f2b7a486640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b740ab920 con 0x7f2b7413f710 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.577+0000 7f2b7a486640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b74135140 con 0x7f2b741430c0 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.577+0000 7f2b7a486640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b74134e10 con 0x7f2b740a98f0 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.577+0000 7f2b7a486640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:30.580 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.577+0000 7f2b7a486640 10 monclient(hunting): authenticate will time out at 7841.864970s 2024-03-25T18:33:30.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.592+0000 7f2b73fff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f2b740a98f0 0x7f2b741385e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:43216/0 (socket says 172.21.15.102:43216) 2024-03-25T18:33:30.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.592+0000 7f2b73fff640 1 -- 172.21.15.102:0/3239392313 learned_addr learned my addr 172.21.15.102:0/3239392313 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:30.594 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.593+0000 7f2b70ff9640 10 client.?.objecter ms_handle_connect 0x7f2b740a98f0 2024-03-25T18:33:30.594 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.593+0000 7f2b70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.596+0000 7f2b70ff9640 10 client.?.objecter ms_handle_connect 0x7f2b7413f710 2024-03-25T18:33:30.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.596+0000 7f2b70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.598+0000 7f2b70ff9640 10 client.?.objecter ms_handle_connect 0x7f2b741430c0 2024-03-25T18:33:30.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.598+0000 7f2b70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:30.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.601+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860447140 0 0) 0x7f2b74134e10 con 0x7f2b740a98f0 2024-03-25T18:33:30.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.601+0000 7f2b70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.601+0000 7f2b70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.601+0000 7f2b70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.601+0000 7f2b70ff9640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:33:30.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.601+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b60004eb0 con 0x7f2b740a98f0 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.604+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535323703 0 0) 0x7f2b74135140 con 0x7f2b741430c0 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.604+0000 7f2b70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.604+0000 7f2b70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.604+0000 7f2b70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.604+0000 7f2b70ff9640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:33:30.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.604+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b74134e10 con 0x7f2b741430c0 2024-03-25T18:33:30.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.607+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552581975 0 0) 0x7f2b740ab920 con 0x7f2b7413f710 2024-03-25T18:33:30.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.607+0000 7f2b70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:33:30.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.607+0000 7f2b70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:30.608 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.607+0000 7f2b70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:30.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.607+0000 7f2b70ff9640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:33:30.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.607+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b74135140 con 0x7f2b7413f710 2024-03-25T18:33:30.617 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.617+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 839396209 0 0) 0x7f2b74135140 con 0x7f2b7413f710 2024-03-25T18:33:30.617 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.617+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b740ab920 con 0x7f2b7413f710 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.617+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2b68001830 con 0x7f2b7413f710 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.617+0000 7f2b70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.617+0000 7f2b70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.617+0000 7f2b70ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:30.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.617+0000 7f2b70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.619+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1905964587 0 0) 0x7f2b60004eb0 con 0x7f2b740a98f0 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.619+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b74135140 con 0x7f2b740a98f0 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.619+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2b640019d0 con 0x7f2b740a98f0 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.619+0000 7f2b70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.619+0000 7f2b70ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.619+0000 7f2b70ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.620 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:30.621 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:30.621 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:30.621 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:30.621 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:30.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.621+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 584242371 0 0) 0x7f2b74134e10 con 0x7f2b741430c0 2024-03-25T18:33:30.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.621+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b60004eb0 con 0x7f2b741430c0 2024-03-25T18:33:30.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.621+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2b6c001e80 con 0x7f2b741430c0 2024-03-25T18:33:30.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.621+0000 7f2b70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.621+0000 7f2b70ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.621+0000 7f2b70ff9640 10 monclient(hunting): dump: 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1329316483 0 0) 0x7f2b740ab920 con 0x7f2b7413f710 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 >> v1:172.21.15.102:6790/0 conn(0x7f2b740a98f0 legacy=0x7f2b741385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 >> v1:172.21.15.146:6789/0 conn(0x7f2b741430c0 legacy=0x7f2b74135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 1 monclient: found mon.a 2024-03-25T18:33:30.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b74137840 con 0x7f2b7413f710 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 10 monclient: _finish_auth 0 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 10 monclient: _renew_subs 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b74137a90 con 0x7f2b7413f710 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 10 monclient: _renew_subs 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b741347e0 con 0x7f2b7413f710 2024-03-25T18:33:30.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.622+0000 7f2b7a486640 1 librados: init done 2024-03-25T18:33:30.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.623+0000 7f2b7a486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.623+0000 7f2b7a486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.623+0000 7f2b7a486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:30.626 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.623+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b74134fa0 con 0x7f2b7413f710 2024-03-25T18:33:30.931 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.930+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b68001d40 con 0x7f2b7413f710 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.930+0000 7f2b70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.930+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2b680035f0 con 0x7f2b7413f710 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.930+0000 7f2b70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.930+0000 7f2b70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.930+0000 7f2b70ff9640 10 monclient: dump: 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:30.932 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:30.933 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:30.933 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:30.933 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:30.933 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.930+0000 7f2b727fc640 4 set_mon_vals no callback set 2024-03-25T18:33:30.933 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.931+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f2b68003810 con 0x7f2b7413f710 2024-03-25T18:33:30.933 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.932+0000 7f2b73fff640 10 monclient: get_auth_request con 0x7f2b6006e0f0 auth_method 0 2024-03-25T18:33:30.933 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.932+0000 7f2b70ff9640 10 client.4572.objecter ms_handle_connect 0x7f2b6006e0f0 2024-03-25T18:33:30.937 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.936+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f2b680533f0 con 0x7f2b7413f710 2024-03-25T18:33:30.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.936+0000 7f2b70ff9640 10 client.4572.objecter ms_dispatch 0x7f2b74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:30.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.936+0000 7f2b70ff9640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:30.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.936+0000 7f2b70ff9640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:30.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.936+0000 7f2b70ff9640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-25T18:33:30.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.936+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b6804ed30 con 0x7f2b7413f710 2024-03-25T18:33:30.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.936+0000 7f2b70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:30.938 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:30.936+0000 7f2b70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:31.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.177+0000 7f2b7a486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:31.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.177+0000 7f2b7a486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:31.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.177+0000 7f2b7a486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:33:31.178 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.177+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b740627b0 con 0x7f2b7413f710 2024-03-25T18:33:31.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.215+0000 7f2b70ff9640 1 -- 172.21.15.102:0/3239392313 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f2b68050e30 con 0x7f2b7413f710 2024-03-25T18:33:31.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.215+0000 7f2b70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:31.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.215+0000 7f2b70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:31.216 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:33:31.216 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:31.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 10 librados: watch_flush enter 2024-03-25T18:33:31.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 10 librados: watch_flush exit 2024-03-25T18:33:31.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-25T18:33:31.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 10 client.4572.objecter successfully canceled tick 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 >> v1:172.21.15.146:6800/3848252714 conn(0x7f2b6006e0f0 legacy=0x7f2b600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 10 monclient: shutdown 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 >> v1:172.21.15.102:6789/0 conn(0x7f2b7413f710 legacy=0x7f2b74133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 shutdown_connections 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 >> 172.21.15.102:0/3239392313 conn(0x7f2b740b7ea0 msgr2=0x7f2b7413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 shutdown_connections 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 1 -- 172.21.15.102:0/3239392313 wait complete. 2024-03-25T18:33:31.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:31.218+0000 7f2b7a486640 1 librados: shutdown 2024-03-25T18:33:31.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:38.039 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:38.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.155+0000 7f270e0d9640 10 monclient: get_monmap_and_config 2024-03-25T18:33:38.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.155+0000 7f270e0d9640 10 monclient: build_initial_monmap 2024-03-25T18:33:38.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.155+0000 7f270e0d9640 10 monclient: monmap: 2024-03-25T18:33:38.156 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:38.156 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:38.156 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:33:38.156468+0000 2024-03-25T18:33:38.156 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:33:38.156468+0000 2024-03-25T18:33:38.157 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:38.157 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:38.157 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:38.157 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:33:38.157 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:33:38.157 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:38.158 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.156+0000 7f270e0d9640 1 Processor -- start 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.156+0000 7f270e0d9640 1 -- start start 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.156+0000 7f270e0d9640 10 monclient: init 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 10 monclient(hunting): picked mon.noname-c con 0x7f27080a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 10 monclient(hunting): picked mon.noname-a con 0x7f270813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 10 monclient(hunting): picked mon.noname-b con 0x7f27081430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2708142520 con 0x7f270813f710 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27080b28a0 con 0x7f27080a98f0 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27080b2a30 con 0x7f27081430c0 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:38.159 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.157+0000 7f270e0d9640 10 monclient(hunting): authenticate will time out at 7849.445741s 2024-03-25T18:33:38.163 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.163+0000 7f2707fff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f27081430c0 0x7f270814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:44658/0 (socket says 172.21.15.102:44658) 2024-03-25T18:33:38.164 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.163+0000 7f2707fff640 1 -- 172.21.15.102:0/2989261160 learned_addr learned my addr 172.21.15.102:0/2989261160 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:38.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.169+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636735173 0 0) 0x7f27080b2a30 con 0x7f27081430c0 2024-03-25T18:33:38.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.169+0000 7f27067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:38.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.169+0000 7f27067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:38.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.169+0000 7f27067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:38.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.169+0000 7f27067fc640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:33:38.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.169+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f0003520 con 0x7f27081430c0 2024-03-25T18:33:38.179 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764416131 0 0) 0x7f26f0003520 con 0x7f27081430c0 2024-03-25T18:33:38.179 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 >> v1:172.21.15.146:6789/0 conn(0x7f27080a98f0 legacy=0x7f27080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.179 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 >> v1:172.21.15.102:6789/0 conn(0x7f270813f710 legacy=0x7f27081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:33:38.179 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:38.179 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 1 monclient: found mon.noname-b 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27080b44c0 con 0x7f27081430c0 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2700001a10 con 0x7f27081430c0 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f270e0d9640 5 monclient: authenticate success, global_id 4550 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f270e0d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f27067fc640 10 monclient: dump: 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:38.180 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:38.181 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:38.181 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:38.181 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:38.181 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:38.181 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:38.181 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:38.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.178+0000 7f270e0d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:38.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.184+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2700001be0 con 0x7f27081430c0 2024-03-25T18:33:38.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.184+0000 7f27067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:38.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.184+0000 7f27067fc640 1 -- 172.21.15.102:0/2989261160 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2700002030 con 0x7f27081430c0 2024-03-25T18:33:38.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.184+0000 7f270e0d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:38.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.184+0000 7f270e0d9640 4 set_mon_vals no callback set 2024-03-25T18:33:38.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.184+0000 7f27067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:38.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.184+0000 7f27067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:38.185 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.184+0000 7f27067fc640 10 monclient: dump: 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:38.186 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:38.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.185+0000 7f270e0d9640 10 monclient: shutdown 2024-03-25T18:33:38.187 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.185+0000 7f270e0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.185+0000 7f270e0d9640 1 -- 172.21.15.102:0/2989261160 >> v1:172.21.15.102:6790/0 conn(0x7f27081430c0 legacy=0x7f270814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.185+0000 7f270e0d9640 1 -- 172.21.15.102:0/2989261160 shutdown_connections 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.185+0000 7f270e0d9640 1 -- 172.21.15.102:0/2989261160 >> 172.21.15.102:0/2989261160 conn(0x7f27080b7ea0 msgr2=0x7f27080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.186+0000 7f270e0d9640 1 -- 172.21.15.102:0/2989261160 shutdown_connections 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.186+0000 7f270e0d9640 1 -- 172.21.15.102:0/2989261160 wait complete. 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.186+0000 7f270e0d9640 10 monclient: build_initial_monmap 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.186+0000 7f270e0d9640 10 monclient: monmap: 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:33:38.188 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 librados: starting msgr at 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 librados: starting objecter 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 Processor -- start 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 -- start start 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 librados: setting wanted keys 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 librados: calling monclient init 2024-03-25T18:33:38.189 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 10 monclient: init 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 10 monclient(hunting): picked mon.noname-a con 0x7f27080a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 10 monclient(hunting): picked mon.noname-c con 0x7f270813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 10 monclient(hunting): picked mon.noname-b con 0x7f27081430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27081d7780 con 0x7f27080a98f0 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27080abd70 con 0x7f27081430c0 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27081d41e0 con 0x7f270813f710 2024-03-25T18:33:38.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:38.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.187+0000 7f270e0d9640 10 monclient(hunting): authenticate will time out at 7849.475919s 2024-03-25T18:33:38.193 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.193+0000 7f2707fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f27081430c0 0x7f27081383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:58576/0 (socket says 172.21.15.102:58576) 2024-03-25T18:33:38.193 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.193+0000 7f2707fff640 1 -- 172.21.15.102:0/3420552630 learned_addr learned my addr 172.21.15.102:0/3420552630 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:38.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.195+0000 7f26effff640 10 client.?.objecter ms_handle_connect 0x7f27081430c0 2024-03-25T18:33:38.196 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.195+0000 7f26effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:38.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.204+0000 7f26effff640 10 client.?.objecter ms_handle_connect 0x7f270813f710 2024-03-25T18:33:38.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.204+0000 7f26effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:38.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.205+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170551636 0 0) 0x7f27080abd70 con 0x7f27081430c0 2024-03-25T18:33:38.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.205+0000 7f26effff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:38.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.205+0000 7f26effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:38.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.205+0000 7f26effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:38.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.205+0000 7f26effff640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:33:38.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.205+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26f4004eb0 con 0x7f27081430c0 2024-03-25T18:33:38.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.209+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051424476 0 0) 0x7f27081d41e0 con 0x7f270813f710 2024-03-25T18:33:38.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.209+0000 7f26effff640 10 monclient(hunting): _negotiate 2024-03-25T18:33:38.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.209+0000 7f26effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:38.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.209+0000 7f26effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:38.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.209+0000 7f26effff640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:33:38.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.209+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27080abd70 con 0x7f270813f710 2024-03-25T18:33:38.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.209+0000 7f26effff640 10 client.?.objecter ms_handle_connect 0x7f27080a98f0 2024-03-25T18:33:38.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.209+0000 7f26effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.219+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 511515977 0 0) 0x7f27080abd70 con 0x7f270813f710 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.219+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27081d41e0 con 0x7f270813f710 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.219+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f26fc001510 con 0x7f270813f710 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.219+0000 7f26effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.219+0000 7f26effff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.219+0000 7f26effff640 10 monclient(hunting): dump: 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:38.220 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:38.221 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:38.221 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.219+0000 7f26effff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:38.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3574575809 0 0) 0x7f27081d41e0 con 0x7f270813f710 2024-03-25T18:33:38.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 >> v1:172.21.15.146:6789/0 conn(0x7f27081430c0 legacy=0x7f27081383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 >> v1:172.21.15.102:6789/0 conn(0x7f27080a98f0 legacy=0x7f27081d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 10 monclient: _finish_hunting 0 2024-03-25T18:33:38.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 1 monclient: found mon.c 2024-03-25T18:33:38.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27081d4370 con 0x7f270813f710 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 10 monclient: _finish_auth 0 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 10 monclient: _check_auth_tickets 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f26effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 10 monclient: _renew_subs 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27081d8370 con 0x7f270813f710 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 10 monclient: _renew_subs 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27081d50a0 con 0x7f270813f710 2024-03-25T18:33:38.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.229+0000 7f270e0d9640 1 librados: init done 2024-03-25T18:33:38.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.230+0000 7f270e0d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:38.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.230+0000 7f270e0d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:38.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.230+0000 7f270e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:38.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.230+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27080b3ef0 con 0x7f270813f710 2024-03-25T18:33:38.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.555+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26fc001a20 con 0x7f270813f710 2024-03-25T18:33:38.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.555+0000 7f26effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:38.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.555+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f26fc003910 con 0x7f270813f710 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.555+0000 7f26effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.555+0000 7f26effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.555+0000 7f26effff640 10 monclient: dump: 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:38.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.555+0000 7f2705ffb640 4 set_mon_vals no callback set 2024-03-25T18:33:38.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.556+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f26fc0198e0 con 0x7f270813f710 2024-03-25T18:33:38.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.556+0000 7f27077fe640 10 monclient: get_auth_request con 0x7f26f406dd90 auth_method 0 2024-03-25T18:33:38.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.557+0000 7f26effff640 10 client.4553.objecter ms_handle_connect 0x7f26f406dd90 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.562+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f26fc053520 con 0x7f270813f710 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.562+0000 7f26effff640 10 client.4553.objecter ms_dispatch 0x7f2708000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.562+0000 7f26effff640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.562+0000 7f26effff640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.562+0000 7f26effff640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.562+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26fc04ee60 con 0x7f270813f710 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.562+0000 7f26effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:38.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.562+0000 7f26effff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:38.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.815+0000 7f270e0d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.815+0000 7f270e0d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.815+0000 7f270e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:33:38.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.815+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27080627b0 con 0x7f270813f710 2024-03-25T18:33:38.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.833+0000 7f26effff640 1 -- 172.21.15.102:0/3420552630 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f26fc050f60 con 0x7f270813f710 2024-03-25T18:33:38.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.833+0000 7f26effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:38.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.833+0000 7f26effff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:38.835 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:33:38.835 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:38.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 10 librados: watch_flush enter 2024-03-25T18:33:38.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 10 librados: watch_flush exit 2024-03-25T18:33:38.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T18:33:38.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 10 client.4553.objecter successfully canceled tick 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 >> v1:172.21.15.146:6800/3848252714 conn(0x7f26f406dd90 legacy=0x7f26f4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 10 monclient: shutdown 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 >> v1:172.21.15.102:6790/0 conn(0x7f270813f710 legacy=0x7f27081d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 shutdown_connections 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 >> 172.21.15.102:0/3420552630 conn(0x7f27080b7ea0 msgr2=0x7f2708074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 shutdown_connections 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 1 -- 172.21.15.102:0/3420552630 wait complete. 2024-03-25T18:33:38.838 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:38.836+0000 7f270e0d9640 1 librados: shutdown 2024-03-25T18:33:38.847 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:44.220 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:00.455 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:44.269 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:00.505 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:44.291 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:00.526 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:44.319 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:00.554 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:44.324 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:00.537 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:44.328 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:00.563 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:44.332 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:00.544 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:44.365 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:00.600 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:44.369 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:00.582 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:33:45.655 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:45.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.766+0000 7f6ac41c1640 10 monclient: get_monmap_and_config 2024-03-25T18:33:45.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.766+0000 7f6ac41c1640 10 monclient: build_initial_monmap 2024-03-25T18:33:45.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.766+0000 7f6ac41c1640 10 monclient: monmap: 2024-03-25T18:33:45.767 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:45.767 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:45.767 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:33:45.767198+0000 2024-03-25T18:33:45.767 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:33:45.767198+0000 2024-03-25T18:33:45.767 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:45.768 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:45.768 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:45.768 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:33:45.768 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:33:45.768 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.767+0000 7f6ac41c1640 1 Processor -- start 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.767+0000 7f6ac41c1640 1 -- start start 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.767+0000 7f6ac41c1640 10 monclient: init 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 10 monclient(hunting): picked mon.noname-c con 0x7f6abc0b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 10 monclient(hunting): picked mon.noname-a con 0x7f6abc0b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 10 monclient(hunting): picked mon.noname-b con 0x7f6abc14be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0af940 con 0x7f6abc0b1730 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0b0a70 con 0x7f6abc0b4d60 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0b0d50 con 0x7f6abc14be90 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:45.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.768+0000 7f6ac41c1640 10 monclient(hunting): authenticate will time out at 7857.056209s 2024-03-25T18:33:45.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.777+0000 7f6ac275e640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f6abc14be90 0x7f6abc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:34262/0 (socket says 172.21.15.102:34262) 2024-03-25T18:33:45.778 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.777+0000 7f6ac275e640 1 -- 172.21.15.102:0/1575103097 learned_addr learned my addr 172.21.15.102:0/1575103097 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:45.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.790+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667748378 0 0) 0x7f6abc0b0d50 con 0x7f6abc14be90 2024-03-25T18:33:45.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.790+0000 7f6ac0f5b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.790+0000 7f6ac0f5b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.790+0000 7f6ac0f5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.790+0000 7f6ac0f5b640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:33:45.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.790+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa8003520 con 0x7f6abc14be90 2024-03-25T18:33:45.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.797+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618822617 0 0) 0x7f6abc0af940 con 0x7f6abc0b1730 2024-03-25T18:33:45.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.797+0000 7f6ac0f5b640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.797+0000 7f6ac0f5b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.797+0000 7f6ac0f5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.797+0000 7f6ac0f5b640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:33:45.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.797+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6abc0b0d50 con 0x7f6abc0b1730 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225207543 0 0) 0x7f6aa8003520 con 0x7f6abc14be90 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 >> v1:172.21.15.146:6789/0 conn(0x7f6abc0b4d60 legacy=0x7f6abc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 >> v1:172.21.15.102:6789/0 conn(0x7f6abc0b1730 legacy=0x7f6abc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 10 monclient: _finish_hunting 0 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 1 monclient: found mon.noname-b 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0b1030 con 0x7f6abc14be90 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 10 monclient: _finish_auth 0 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 10 monclient: _check_auth_tickets 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6ab4001a10 con 0x7f6abc14be90 2024-03-25T18:33:45.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac0f5b640 10 monclient: dump: 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac41c1640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:33:45.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.799+0000 7f6ac41c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:45.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.808+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ab4001b70 con 0x7f6abc14be90 2024-03-25T18:33:45.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.808+0000 7f6ac0f5b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:45.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.808+0000 7f6ac0f5b640 1 -- 172.21.15.102:0/1575103097 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6ab4001f80 con 0x7f6abc14be90 2024-03-25T18:33:45.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.808+0000 7f6ac0f5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.808+0000 7f6ac0f5b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:45.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.808+0000 7f6ac0f5b640 10 monclient: dump: 2024-03-25T18:33:45.809 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:45.809 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.808+0000 7f6ac41c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:45.810 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.808+0000 7f6ac41c1640 4 set_mon_vals no callback set 2024-03-25T18:33:45.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.809+0000 7f6ac41c1640 10 monclient: shutdown 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.809+0000 7f6ac41c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.809+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1575103097 >> v1:172.21.15.102:6790/0 conn(0x7f6abc14be90 legacy=0x7f6abc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.809+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1575103097 shutdown_connections 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.809+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1575103097 >> 172.21.15.102:0/1575103097 conn(0x7f6abc0617a0 msgr2=0x7f6abc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.809+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1575103097 shutdown_connections 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.809+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1575103097 wait complete. 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.810+0000 7f6ac41c1640 10 monclient: build_initial_monmap 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.810+0000 7f6ac41c1640 10 monclient: monmap: 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:33:45.813 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.810+0000 7f6ac41c1640 1 librados: starting msgr at 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.810+0000 7f6ac41c1640 1 librados: starting objecter 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.810+0000 7f6ac41c1640 1 Processor -- start 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 1 -- start start 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 1 librados: setting wanted keys 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 1 librados: calling monclient init 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 10 monclient: init 2024-03-25T18:33:45.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 10 monclient(hunting): picked mon.noname-c con 0x7f6abc0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 10 monclient(hunting): picked mon.noname-b con 0x7f6abc0b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 10 monclient(hunting): picked mon.noname-a con 0x7f6abc14be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc1db2f0 con 0x7f6abc14be90 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc13a530 con 0x7f6abc0b4d60 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc1daea0 con 0x7f6abc0b1730 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:45.815 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.811+0000 7f6ac41c1640 10 monclient(hunting): authenticate will time out at 7857.099850s 2024-03-25T18:33:45.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.820+0000 7f6ac175c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6abc0b4d60 0x7f6abc13aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:42946/0 (socket says 172.21.15.102:42946) 2024-03-25T18:33:45.820 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.820+0000 7f6ac175c640 1 -- 172.21.15.102:0/1214289790 learned_addr learned my addr 172.21.15.102:0/1214289790 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.823+0000 7f6ab27fc640 10 client.?.objecter ms_handle_connect 0x7f6abc0b4d60 2024-03-25T18:33:45.824 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.823+0000 7f6ab27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.831+0000 7f6ab27fc640 10 client.?.objecter ms_handle_connect 0x7f6abc14be90 2024-03-25T18:33:45.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.831+0000 7f6ab27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.833+0000 7f6ab27fc640 10 client.?.objecter ms_handle_connect 0x7f6abc0b1730 2024-03-25T18:33:45.834 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.833+0000 7f6ab27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:45.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.834+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654220097 0 0) 0x7f6abc13a530 con 0x7f6abc0b4d60 2024-03-25T18:33:45.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.834+0000 7f6ab27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.834+0000 7f6ab27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.834+0000 7f6ab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.834+0000 7f6ab27fc640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:33:45.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.835+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa0004eb0 con 0x7f6abc0b4d60 2024-03-25T18:33:45.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.843+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602725766 0 0) 0x7f6abc1db2f0 con 0x7f6abc14be90 2024-03-25T18:33:45.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.843+0000 7f6ab27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.843+0000 7f6ab27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.843+0000 7f6ab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.843+0000 7f6ab27fc640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:33:45.844 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.843+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6abc13a530 con 0x7f6abc14be90 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.845+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3894741666 0 0) 0x7f6aa0004eb0 con 0x7f6abc0b4d60 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.846+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6abc1db2f0 con 0x7f6abc0b4d60 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.846+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6aac001a10 con 0x7f6abc0b4d60 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.846+0000 7f6ab27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.846+0000 7f6ab27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.846+0000 7f6ab27fc640 10 monclient(hunting): dump: 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:45.847 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:45.848 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:45.848 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:45.848 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:45.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.846+0000 7f6ab27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:45.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.851+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771690869 0 0) 0x7f6abc1daea0 con 0x7f6abc0b1730 2024-03-25T18:33:45.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.851+0000 7f6ab27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:33:45.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.851+0000 7f6ab27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:45.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.851+0000 7f6ab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:45.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.851+0000 7f6ab27fc640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:33:45.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.851+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa0004eb0 con 0x7f6abc0b1730 2024-03-25T18:33:45.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 373404392 0 0) 0x7f6abc1db2f0 con 0x7f6abc0b4d60 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 >> v1:172.21.15.102:6790/0 conn(0x7f6abc0b1730 legacy=0x7f6abc1db780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 >> v1:172.21.15.102:6789/0 conn(0x7f6abc14be90 legacy=0x7f6abc13b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 1 monclient: found mon.b 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc1db030 con 0x7f6abc0b4d60 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 10 monclient: _finish_auth 0 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ab27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 10 monclient: _renew_subs 2024-03-25T18:33:45.862 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:45.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc139140 con 0x7f6abc0b4d60 2024-03-25T18:33:45.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:45.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 10 monclient: _renew_subs 2024-03-25T18:33:45.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:45.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc13bac0 con 0x7f6abc0b4d60 2024-03-25T18:33:45.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.860+0000 7f6ac41c1640 1 librados: init done 2024-03-25T18:33:45.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.861+0000 7f6ac41c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:45.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.861+0000 7f6ac41c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:45.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.861+0000 7f6ac41c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:45.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:45.861+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc0627b0 con 0x7f6abc0b4d60 2024-03-25T18:33:46.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.133+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6aac001c50 con 0x7f6abc0b4d60 2024-03-25T18:33:46.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.133+0000 7f6ab27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:46.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.133+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6aac003a60 con 0x7f6abc0b4d60 2024-03-25T18:33:46.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.133+0000 7f6ab27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:46.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.133+0000 7f6ab27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.133+0000 7f6ab27fc640 10 monclient: dump: 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:46.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.133+0000 7f6ab3fff640 4 set_mon_vals no callback set 2024-03-25T18:33:46.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.134+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f6aac003c80 con 0x7f6abc0b4d60 2024-03-25T18:33:46.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.135+0000 7f6ac1f5d640 10 monclient: get_auth_request con 0x7f6aa006de00 auth_method 0 2024-03-25T18:33:46.136 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.135+0000 7f6ab27fc640 10 client.4582.objecter ms_handle_connect 0x7f6aa006de00 2024-03-25T18:33:46.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.140+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f6aac054880 con 0x7f6abc0b4d60 2024-03-25T18:33:46.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.140+0000 7f6ab27fc640 10 client.4582.objecter ms_dispatch 0x7f6abc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:46.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.140+0000 7f6ab27fc640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:46.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.140+0000 7f6ab27fc640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:46.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.140+0000 7f6ab27fc640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:33:46.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.140+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6aac050390 con 0x7f6abc0b4d60 2024-03-25T18:33:46.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.140+0000 7f6ab27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:46.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.140+0000 7f6ab27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:46.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.371+0000 7f6ac41c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.371+0000 7f6ac41c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.372+0000 7f6ac41c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:46.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.372+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6abc069f40 con 0x7f6abc0b4d60 2024-03-25T18:33:46.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.431+0000 7f6ab27fc640 1 -- 172.21.15.102:0/1214289790 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f6aac0522c0 con 0x7f6abc0b4d60 2024-03-25T18:33:46.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.431+0000 7f6ab27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:46.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.431+0000 7f6ab27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:46.432 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:33:46.433 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:46.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 10 librados: watch_flush enter 2024-03-25T18:33:46.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 10 librados: watch_flush exit 2024-03-25T18:33:46.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 >> v1:172.21.15.146:6800/3848252714 conn(0x7f6aa006de00 legacy=0x7f6aa00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 10 monclient: shutdown 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 >> v1:172.21.15.146:6789/0 conn(0x7f6abc0b4d60 legacy=0x7f6abc13aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 shutdown_connections 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 >> 172.21.15.102:0/1214289790 conn(0x7f6abc0617a0 msgr2=0x7f6abc0b69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 shutdown_connections 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 1 -- 172.21.15.102:0/1214289790 wait complete. 2024-03-25T18:33:46.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:46.434+0000 7f6ac41c1640 1 librados: shutdown 2024-03-25T18:33:46.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:33:53.254 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:33:53.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.365+0000 7fe59069a640 10 monclient: get_monmap_and_config 2024-03-25T18:33:53.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.365+0000 7fe59069a640 10 monclient: build_initial_monmap 2024-03-25T18:33:53.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.365+0000 7fe59069a640 10 monclient: monmap: 2024-03-25T18:33:53.367 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:53.367 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:53.368 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:33:53.367396+0000 2024-03-25T18:33:53.368 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:33:53.367396+0000 2024-03-25T18:33:53.368 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:53.368 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:53.368 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:53.368 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:33:53.368 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:33:53.368 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:53.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.366+0000 7fe59069a640 1 Processor -- start 2024-03-25T18:33:53.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.366+0000 7fe59069a640 1 -- start start 2024-03-25T18:33:53.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.366+0000 7fe59069a640 10 monclient: init 2024-03-25T18:33:53.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:53.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:33:53.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 10 monclient(hunting): picked mon.noname-c con 0x7fe5880a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:53.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 10 monclient(hunting): picked mon.noname-b con 0x7fe58813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:53.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 10 monclient(hunting): picked mon.noname-a con 0x7fe5881430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:53.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe588142520 con 0x7fe5881430c0 2024-03-25T18:33:53.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5880b28a0 con 0x7fe5880a98f0 2024-03-25T18:33:53.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5880b2a30 con 0x7fe58813f710 2024-03-25T18:33:53.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:53.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.367+0000 7fe59069a640 10 monclient(hunting): authenticate will time out at 7864.656535s 2024-03-25T18:33:53.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.377+0000 7fe58e436640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe5880a98f0 0x7fe5880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:58352/0 (socket says 172.21.15.102:58352) 2024-03-25T18:33:53.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.377+0000 7fe58e436640 1 -- 172.21.15.102:0/3378207249 learned_addr learned my addr 172.21.15.102:0/3378207249 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:53.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.395+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950439966 0 0) 0x7fe5880b28a0 con 0x7fe5880a98f0 2024-03-25T18:33:53.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.395+0000 7fe58d434640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.395+0000 7fe58d434640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.395+0000 7fe58d434640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.395+0000 7fe58d434640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:33:53.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.395+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe574003520 con 0x7fe5880a98f0 2024-03-25T18:33:53.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.398+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396841084 0 0) 0x7fe5880b2a30 con 0x7fe58813f710 2024-03-25T18:33:53.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.398+0000 7fe58d434640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.398+0000 7fe58d434640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.398+0000 7fe58d434640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.398+0000 7fe58d434640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:33:53.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.398+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5880b28a0 con 0x7fe58813f710 2024-03-25T18:33:53.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 97216641 0 0) 0x7fe574003520 con 0x7fe5880a98f0 2024-03-25T18:33:53.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 >> v1:172.21.15.102:6790/0 conn(0x7fe58813f710 legacy=0x7fe5881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 >> v1:172.21.15.102:6789/0 conn(0x7fe5881430c0 legacy=0x7fe58814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 10 monclient: _finish_hunting 0 2024-03-25T18:33:53.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 1 monclient: found mon.noname-c 2024-03-25T18:33:53.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:53.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:33:53.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5880b44c0 con 0x7fe5880a98f0 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 10 monclient: _finish_auth 0 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 10 monclient: _check_auth_tickets 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe584001690 con 0x7fe5880a98f0 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe58d434640 10 monclient: dump: 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:53.410 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:53.411 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:53.411 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:53.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe59069a640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:33:53.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.407+0000 7fe59069a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:33:53.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.425+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5840017f0 con 0x7fe5880a98f0 2024-03-25T18:33:53.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.425+0000 7fe58d434640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:53.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.425+0000 7fe58d434640 1 -- 172.21.15.102:0/3378207249 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe584001c00 con 0x7fe5880a98f0 2024-03-25T18:33:53.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.425+0000 7fe58d434640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.425+0000 7fe58d434640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.425+0000 7fe58d434640 10 monclient: dump: 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.425+0000 7fe59069a640 10 monclient: get_monmap_and_config success 2024-03-25T18:33:53.428 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.425+0000 7fe59069a640 4 set_mon_vals no callback set 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.426+0000 7fe59069a640 10 monclient: shutdown 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.426+0000 7fe59069a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.426+0000 7fe59069a640 1 -- 172.21.15.102:0/3378207249 >> v1:172.21.15.146:6789/0 conn(0x7fe5880a98f0 legacy=0x7fe5880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.427+0000 7fe59069a640 1 -- 172.21.15.102:0/3378207249 shutdown_connections 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.427+0000 7fe59069a640 1 -- 172.21.15.102:0/3378207249 >> 172.21.15.102:0/3378207249 conn(0x7fe5880b7ea0 msgr2=0x7fe5880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.427+0000 7fe59069a640 1 -- 172.21.15.102:0/3378207249 shutdown_connections 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.427+0000 7fe59069a640 1 -- 172.21.15.102:0/3378207249 wait complete. 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.427+0000 7fe59069a640 10 monclient: build_initial_monmap 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.427+0000 7fe59069a640 10 monclient: monmap: 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:33:53.431 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 1 librados: starting msgr at 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 1 librados: starting objecter 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 1 Processor -- start 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 1 -- start start 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 1 librados: setting wanted keys 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 1 librados: calling monclient init 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 10 monclient: init 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 10 monclient: _reopen_session rank -1 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:33:53.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 10 monclient(hunting): picked mon.noname-a con 0x7fe5880a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:33:53.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.428+0000 7fe59069a640 10 monclient(hunting): picked mon.noname-c con 0x7fe58813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:33:53.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.429+0000 7fe59069a640 10 monclient(hunting): picked mon.noname-b con 0x7fe5881430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:33:53.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.429+0000 7fe59069a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe588135730 con 0x7fe5880a98f0 2024-03-25T18:33:53.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.429+0000 7fe59069a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5881358c0 con 0x7fe5881430c0 2024-03-25T18:33:53.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.429+0000 7fe59069a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe588135a50 con 0x7fe58813f710 2024-03-25T18:33:53.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.429+0000 7fe59069a640 10 monclient(hunting): _renew_subs 2024-03-25T18:33:53.433 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.429+0000 7fe59069a640 10 monclient(hunting): authenticate will time out at 7864.718056s 2024-03-25T18:33:53.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.434+0000 7fe58ec37640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe5881430c0 0x7fe5881d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:58402/0 (socket says 172.21.15.102:58402) 2024-03-25T18:33:53.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.434+0000 7fe58ec37640 1 -- 172.21.15.102:0/3292102283 learned_addr learned my addr 172.21.15.102:0/3292102283 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:33:53.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.435+0000 7fe57effd640 10 client.?.objecter ms_handle_connect 0x7fe5881430c0 2024-03-25T18:33:53.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.435+0000 7fe57effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.438+0000 7fe57effd640 10 client.?.objecter ms_handle_connect 0x7fe58813f710 2024-03-25T18:33:53.440 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.438+0000 7fe57effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.442+0000 7fe57effd640 10 client.?.objecter ms_handle_connect 0x7fe5880a98f0 2024-03-25T18:33:53.443 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.442+0000 7fe57effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:33:53.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.448+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252560573 0 0) 0x7fe5881358c0 con 0x7fe5881430c0 2024-03-25T18:33:53.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.448+0000 7fe57effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.448+0000 7fe57effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.448+0000 7fe57effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.448+0000 7fe57effd640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:33:53.450 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.448+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe56c004eb0 con 0x7fe5881430c0 2024-03-25T18:33:53.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.449+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447984227 0 0) 0x7fe588135a50 con 0x7fe58813f710 2024-03-25T18:33:53.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.449+0000 7fe57effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.451 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.449+0000 7fe57effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.449+0000 7fe57effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.449+0000 7fe57effd640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:33:53.452 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.449+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5881358c0 con 0x7fe58813f710 2024-03-25T18:33:53.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.461+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1417471479 0 0) 0x7fe56c004eb0 con 0x7fe5881430c0 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.461+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe588135a50 con 0x7fe5881430c0 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.461+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe580001c30 con 0x7fe5881430c0 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.461+0000 7fe57effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.461+0000 7fe57effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.461+0000 7fe57effd640 10 monclient(hunting): dump: 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:53.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.461+0000 7fe57effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:33:53.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.464+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377592350 0 0) 0x7fe588135730 con 0x7fe5880a98f0 2024-03-25T18:33:53.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.464+0000 7fe57effd640 10 monclient(hunting): _negotiate 2024-03-25T18:33:53.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.464+0000 7fe57effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:33:53.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.464+0000 7fe57effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:33:53.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.464+0000 7fe57effd640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:33:53.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.464+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe56c004eb0 con 0x7fe5880a98f0 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.468+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 598380289 0 0) 0x7fe5881358c0 con 0x7fe58813f710 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.468+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe588135730 con 0x7fe58813f710 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.468+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe578001930 con 0x7fe58813f710 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.468+0000 7fe57effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.468+0000 7fe57effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.468+0000 7fe57effd640 10 monclient(hunting): dump: 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:53.470 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.471 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.471 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.471 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:53.471 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:53.471 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:53.471 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:53.471 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:53.477 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1628038304 0 0) 0x7fe588135a50 con 0x7fe5881430c0 2024-03-25T18:33:53.477 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 >> v1:172.21.15.102:6790/0 conn(0x7fe58813f710 legacy=0x7fe5881d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 >> v1:172.21.15.102:6789/0 conn(0x7fe5880a98f0 legacy=0x7fe5881dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 10 monclient: _finish_hunting 0 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 1 monclient: found mon.b 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5881d5000 con 0x7fe5881430c0 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 10 monclient: _finish_auth 0 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 10 monclient: _check_auth_tickets 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe57effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 10 monclient: _renew_subs 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5881d52b0 con 0x7fe5881430c0 2024-03-25T18:33:53.478 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:33:53.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 10 monclient: _renew_subs 2024-03-25T18:33:53.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:53.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58813c900 con 0x7fe5881430c0 2024-03-25T18:33:53.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.475+0000 7fe59069a640 1 librados: init done 2024-03-25T18:33:53.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.476+0000 7fe59069a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.476+0000 7fe59069a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.479 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.476+0000 7fe59069a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:53.480 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.476+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5881371c0 con 0x7fe5881430c0 2024-03-25T18:33:53.747 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.745+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe580003970 con 0x7fe5881430c0 2024-03-25T18:33:53.747 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.745+0000 7fe57effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.745+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe580003d00 con 0x7fe5881430c0 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.745+0000 7fe57effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.745+0000 7fe57effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.745+0000 7fe57effd640 10 monclient: dump: 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:33:53.748 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:33:53.749 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:33:53.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.745+0000 7fe58cc33640 4 set_mon_vals no callback set 2024-03-25T18:33:53.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.746+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fe580003f20 con 0x7fe5881430c0 2024-03-25T18:33:53.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.747+0000 7fe58e436640 10 monclient: get_auth_request con 0x7fe56c06dfb0 auth_method 0 2024-03-25T18:33:53.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.747+0000 7fe57effd640 10 client.4588.objecter ms_handle_connect 0x7fe56c06dfb0 2024-03-25T18:33:53.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.752+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fe5800549e0 con 0x7fe5881430c0 2024-03-25T18:33:53.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.753+0000 7fe57effd640 10 client.4588.objecter ms_dispatch 0x7fe588000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:33:53.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.753+0000 7fe57effd640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:33:53.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.753+0000 7fe57effd640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:33:53.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.753+0000 7fe57effd640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T18:33:53.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.753+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5800504f0 con 0x7fe5881430c0 2024-03-25T18:33:53.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.753+0000 7fe57effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:33:53.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:53.753+0000 7fe57effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:33:54.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.012+0000 7fe59069a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:54.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.012+0000 7fe59069a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:54.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.012+0000 7fe59069a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:33:54.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.012+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5880627b0 con 0x7fe5881430c0 2024-03-25T18:33:54.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.031+0000 7fe57effd640 1 -- 172.21.15.102:0/3292102283 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fe580052420 con 0x7fe5881430c0 2024-03-25T18:33:54.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.031+0000 7fe57effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:33:54.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.031+0000 7fe57effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:33:54.033 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:33:54.033 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:33:54.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 10 librados: watch_flush enter 2024-03-25T18:33:54.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 10 librados: watch_flush exit 2024-03-25T18:33:54.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T18:33:54.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 10 client.4588.objecter successfully canceled tick 2024-03-25T18:33:54.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 >> v1:172.21.15.146:6800/3848252714 conn(0x7fe56c06dfb0 legacy=0x7fe56c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:54.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 10 monclient: shutdown 2024-03-25T18:33:54.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:33:54.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 >> v1:172.21.15.146:6789/0 conn(0x7fe5881430c0 legacy=0x7fe5881d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:33:54.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 shutdown_connections 2024-03-25T18:33:54.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 >> 172.21.15.102:0/3292102283 conn(0x7fe5880b7ea0 msgr2=0x7fe58814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:33:54.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.033+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 shutdown_connections 2024-03-25T18:33:54.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.034+0000 7fe59069a640 1 -- 172.21.15.102:0/3292102283 wait complete. 2024-03-25T18:33:54.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:33:54.034+0000 7fe59069a640 1 librados: shutdown 2024-03-25T18:33:54.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:00.853 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:00.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.965+0000 7f48c0d8f640 10 monclient: get_monmap_and_config 2024-03-25T18:34:00.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.965+0000 7f48c0d8f640 10 monclient: build_initial_monmap 2024-03-25T18:34:00.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.965+0000 7f48c0d8f640 10 monclient: monmap: 2024-03-25T18:34:00.967 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:00.967 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:00.967 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:34:00.967230+0000 2024-03-25T18:34:00.967 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:34:00.967230+0000 2024-03-25T18:34:00.968 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:00.968 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:00.968 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:00.968 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:34:00.968 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:34:00.968 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:00.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.966+0000 7f48c0d8f640 1 Processor -- start 2024-03-25T18:34:00.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.966+0000 7f48c0d8f640 1 -- start start 2024-03-25T18:34:00.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.966+0000 7f48c0d8f640 10 monclient: init 2024-03-25T18:34:00.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.966+0000 7f48c0d8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:00.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.966+0000 7f48c0d8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:00.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.966+0000 7f48c0d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f48bc0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:00.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.966+0000 7f48c0d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f48bc13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:00.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.967+0000 7f48c0d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f48bc1430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:00.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.967+0000 7f48c0d8f640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc142520 con 0x7f48bc0a98f0 2024-03-25T18:34:00.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.967+0000 7f48c0d8f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b28a0 con 0x7f48bc1430c0 2024-03-25T18:34:00.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.967+0000 7f48c0d8f640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b2a30 con 0x7f48bc13f710 2024-03-25T18:34:00.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.967+0000 7f48c0d8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:00.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.967+0000 7f48c0d8f640 10 monclient(hunting): authenticate will time out at 7872.255993s 2024-03-25T18:34:00.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.969+0000 7f48bad9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f48bc1430c0 0x7f48bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:35032/0 (socket says 172.21.15.102:35032) 2024-03-25T18:34:00.970 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.969+0000 7f48bad9d640 1 -- 172.21.15.102:0/3128572813 learned_addr learned my addr 172.21.15.102:0/3128572813 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:00.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.979+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302870855 0 0) 0x7f48bc0b28a0 con 0x7f48bc1430c0 2024-03-25T18:34:00.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.979+0000 7f48b959a640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.979+0000 7f48b959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.979+0000 7f48b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.979+0000 7f48b959a640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:34:00.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.979+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a4003520 con 0x7f48bc1430c0 2024-03-25T18:34:00.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.987+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730285126 0 0) 0x7f48bc142520 con 0x7f48bc0a98f0 2024-03-25T18:34:00.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.988+0000 7f48b959a640 10 monclient(hunting): _negotiate 2024-03-25T18:34:00.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.988+0000 7f48b959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:00.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.988+0000 7f48b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:00.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.988+0000 7f48b959a640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:34:00.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.988+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48bc0b28a0 con 0x7f48bc0a98f0 2024-03-25T18:34:00.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 769522670 0 0) 0x7f48a4003520 con 0x7f48bc1430c0 2024-03-25T18:34:00.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 >> v1:172.21.15.102:6790/0 conn(0x7f48bc13f710 legacy=0x7f48bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 >> v1:172.21.15.102:6789/0 conn(0x7f48bc0a98f0 legacy=0x7f48bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:00.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 10 monclient: _finish_hunting 0 2024-03-25T18:34:00.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 1 monclient: found mon.noname-c 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0b44c0 con 0x7f48bc1430c0 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 10 monclient: _finish_auth 0 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 10 monclient: _check_auth_tickets 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f48b4001a10 con 0x7f48bc1430c0 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48b959a640 10 monclient: dump: 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:00.999 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:01.000 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:01.000 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:01.000 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:01.000 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:01.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48c0d8f640 5 monclient: authenticate success, global_id 4591 2024-03-25T18:34:01.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:00.996+0000 7f48c0d8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:01.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48b4001b70 con 0x7f48bc1430c0 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48b959a640 1 -- 172.21.15.102:0/3128572813 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f48b4001f80 con 0x7f48bc1430c0 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48c0d8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48c0d8f640 4 set_mon_vals no callback set 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48b959a640 10 monclient: dump: 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:01.006 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:01.007 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:01.007 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:01.007 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:01.007 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:01.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48c0d8f640 10 monclient: shutdown 2024-03-25T18:34:01.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48c0d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:01.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.004+0000 7f48c0d8f640 1 -- 172.21.15.102:0/3128572813 >> v1:172.21.15.146:6789/0 conn(0x7f48bc1430c0 legacy=0x7f48bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.005+0000 7f48c0d8f640 1 -- 172.21.15.102:0/3128572813 shutdown_connections 2024-03-25T18:34:01.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.005+0000 7f48c0d8f640 1 -- 172.21.15.102:0/3128572813 >> 172.21.15.102:0/3128572813 conn(0x7f48bc0b7ea0 msgr2=0x7f48bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:01.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.005+0000 7f48c0d8f640 1 -- 172.21.15.102:0/3128572813 shutdown_connections 2024-03-25T18:34:01.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.005+0000 7f48c0d8f640 1 -- 172.21.15.102:0/3128572813 wait complete. 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 10 monclient: build_initial_monmap 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 10 monclient: monmap: 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 1 librados: starting msgr at 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 1 librados: starting objecter 2024-03-25T18:34:01.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 1 Processor -- start 2024-03-25T18:34:01.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 1 -- start start 2024-03-25T18:34:01.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 1 librados: setting wanted keys 2024-03-25T18:34:01.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 1 librados: calling monclient init 2024-03-25T18:34:01.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.006+0000 7f48c0d8f640 10 monclient: init 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f48bc0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f48bc13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f48bc1430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc06f070 con 0x7f48bc0a98f0 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc14d570 con 0x7f48bc1430c0 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc139a10 con 0x7f48bc13f710 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:01.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.007+0000 7f48c0d8f640 10 monclient(hunting): authenticate will time out at 7872.296539s 2024-03-25T18:34:01.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.016+0000 7f48bad9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f48bc1430c0 0x7f48bc06e5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:35070/0 (socket says 172.21.15.102:35070) 2024-03-25T18:34:01.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.016+0000 7f48bad9d640 1 -- 172.21.15.102:0/93214988 learned_addr learned my addr 172.21.15.102:0/93214988 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:01.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.017+0000 7f48aaffd640 10 client.?.objecter ms_handle_connect 0x7f48bc1430c0 2024-03-25T18:34:01.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.017+0000 7f48aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:01.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.017+0000 7f48aaffd640 10 client.?.objecter ms_handle_connect 0x7f48bc13f710 2024-03-25T18:34:01.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.017+0000 7f48aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:01.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.017+0000 7f48aaffd640 10 client.?.objecter ms_handle_connect 0x7f48bc0a98f0 2024-03-25T18:34:01.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.017+0000 7f48aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:01.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.023+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212187939 0 0) 0x7f48bc139a10 con 0x7f48bc13f710 2024-03-25T18:34:01.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.023+0000 7f48aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:01.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.023+0000 7f48aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:01.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.023+0000 7f48aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:01.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.023+0000 7f48aaffd640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:34:01.025 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.023+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f489c004eb0 con 0x7f48bc13f710 2024-03-25T18:34:01.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.027+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973547866 0 0) 0x7f48bc14d570 con 0x7f48bc1430c0 2024-03-25T18:34:01.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.027+0000 7f48aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:01.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.027+0000 7f48aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:01.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.027+0000 7f48aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:01.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.027+0000 7f48aaffd640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:34:01.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.027+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48bc139a10 con 0x7f48bc1430c0 2024-03-25T18:34:01.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.033+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 216532157 0 0) 0x7f489c004eb0 con 0x7f48bc13f710 2024-03-25T18:34:01.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.033+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48bc14d570 con 0x7f48bc13f710 2024-03-25T18:34:01.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.034+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372286350 0 0) 0x7f48bc06f070 con 0x7f48bc0a98f0 2024-03-25T18:34:01.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.034+0000 7f48aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:01.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.034+0000 7f48aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:01.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.034+0000 7f48aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:01.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.034+0000 7f48aaffd640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:34:01.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.034+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f489c004eb0 con 0x7f48bc0a98f0 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.037+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f48b00019d0 con 0x7f48bc13f710 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.037+0000 7f48aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.037+0000 7f48aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.037+0000 7f48aaffd640 10 monclient(hunting): dump: 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:01.039 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:01.040 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:01.040 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:01.040 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.037+0000 7f48aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.047+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 255933360 0 0) 0x7f48bc139a10 con 0x7f48bc1430c0 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.047+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48bc06f070 con 0x7f48bc1430c0 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.047+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f48b4001e80 con 0x7f48bc1430c0 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.047+0000 7f48aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.047+0000 7f48aaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.047+0000 7f48aaffd640 10 monclient(hunting): dump: 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:01.049 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.050 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.050 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:01.050 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:01.050 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:01.050 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:01.050 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:01.050 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:01.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.049+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4040867728 0 0) 0x7f489c004eb0 con 0x7f48bc0a98f0 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.049+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48bc139a10 con 0x7f48bc0a98f0 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.050+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f48ac001930 con 0x7f48bc0a98f0 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.050+0000 7f48aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.050+0000 7f48aaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.050+0000 7f48aaffd640 10 monclient(hunting): dump: 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:01.052 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:01.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.058+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 708450112 0 0) 0x7f48bc14d570 con 0x7f48bc13f710 2024-03-25T18:34:01.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 >> v1:172.21.15.146:6789/0 conn(0x7f48bc1430c0 legacy=0x7f48bc06e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 >> v1:172.21.15.102:6789/0 conn(0x7f48bc0a98f0 legacy=0x7f48bc14bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:01.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 1 monclient: found mon.c 2024-03-25T18:34:01.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:01.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:01.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc14e800 con 0x7f48bc13f710 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 10 monclient: _renew_subs 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc06fcc0 con 0x7f48bc13f710 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 10 monclient: _renew_subs 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc070050 con 0x7f48bc13f710 2024-03-25T18:34:01.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.059+0000 7f48c0d8f640 1 librados: init done 2024-03-25T18:34:01.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.060+0000 7f48c0d8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:01.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.060+0000 7f48c0d8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:01.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.060+0000 7f48c0d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:01.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.060+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc06ee30 con 0x7f48bc13f710 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.381+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48b00035f0 con 0x7f48bc13f710 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.381+0000 7f48aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.381+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f48b0003a00 con 0x7f48bc13f710 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.381+0000 7f48aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.381+0000 7f48aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.381+0000 7f48aaffd640 10 monclient: dump: 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.383 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:01.384 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:01.384 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:01.384 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:01.384 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:01.384 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:01.384 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:01.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.381+0000 7f48b8d99640 4 set_mon_vals no callback set 2024-03-25T18:34:01.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.382+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f48b0003c20 con 0x7f48bc13f710 2024-03-25T18:34:01.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.382+0000 7f48ba59c640 10 monclient: get_auth_request con 0x7f489c06e120 auth_method 0 2024-03-25T18:34:01.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.383+0000 7f48aaffd640 10 client.4571.objecter ms_handle_connect 0x7f489c06e120 2024-03-25T18:34:01.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.391+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f48b0053560 con 0x7f48bc13f710 2024-03-25T18:34:01.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.391+0000 7f48aaffd640 10 client.4571.objecter ms_dispatch 0x7f48bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:01.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.391+0000 7f48aaffd640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:01.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.391+0000 7f48aaffd640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:01.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.391+0000 7f48aaffd640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-25T18:34:01.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.391+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48b004f070 con 0x7f48bc13f710 2024-03-25T18:34:01.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.391+0000 7f48aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:01.393 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.391+0000 7f48aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:01.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.612+0000 7f48c0d8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.612+0000 7f48c0d8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.612+0000 7f48c0d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:01.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.612+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48bc0627b0 con 0x7f48bc13f710 2024-03-25T18:34:01.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.651+0000 7f48aaffd640 1 -- 172.21.15.102:0/93214988 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f48b0050fa0 con 0x7f48bc13f710 2024-03-25T18:34:01.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.651+0000 7f48aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:01.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.651+0000 7f48aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:01.653 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:34:01.653 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:01.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 10 librados: watch_flush enter 2024-03-25T18:34:01.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 10 librados: watch_flush exit 2024-03-25T18:34:01.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-25T18:34:01.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 10 client.4571.objecter successfully canceled tick 2024-03-25T18:34:01.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 >> v1:172.21.15.146:6800/3848252714 conn(0x7f489c06e120 legacy=0x7f489c0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 10 monclient: shutdown 2024-03-25T18:34:01.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:01.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 >> v1:172.21.15.102:6790/0 conn(0x7f48bc13f710 legacy=0x7f48bc06dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:01.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 shutdown_connections 2024-03-25T18:34:01.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 >> 172.21.15.102:0/93214988 conn(0x7f48bc0b7ea0 msgr2=0x7f48bc13ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:01.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 shutdown_connections 2024-03-25T18:34:01.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 1 -- 172.21.15.102:0/93214988 wait complete. 2024-03-25T18:34:01.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:01.653+0000 7f48c0d8f640 1 librados: shutdown 2024-03-25T18:34:01.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:04.423 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:20.659 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:04.428 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:20.664 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:04.479 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:20.715 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:04.484 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:20.720 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:04.498 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:20.734 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:04.520 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:20.756 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:04.545 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:20.758 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:04.549 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:20.762 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:04.580 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:20.793 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:08.474 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:08.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.589+0000 7f6331fc6640 10 monclient: get_monmap_and_config 2024-03-25T18:34:08.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.589+0000 7f6331fc6640 10 monclient: build_initial_monmap 2024-03-25T18:34:08.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.589+0000 7f6331fc6640 10 monclient: monmap: 2024-03-25T18:34:08.590 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:08.590 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:08.590 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:34:08.590333+0000 2024-03-25T18:34:08.591 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:34:08.590333+0000 2024-03-25T18:34:08.591 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:08.591 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:08.591 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:08.591 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:34:08.591 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:34:08.591 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.590+0000 7f6331fc6640 1 Processor -- start 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.590+0000 7f6331fc6640 1 -- start start 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.590+0000 7f6331fc6640 10 monclient: init 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f632c0b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f632c0b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f632c14be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c0af940 con 0x7f632c14be90 2024-03-25T18:34:08.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c0b0a70 con 0x7f632c0b1730 2024-03-25T18:34:08.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c0b0d50 con 0x7f632c0b4d60 2024-03-25T18:34:08.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:08.593 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.591+0000 7f6331fc6640 10 monclient(hunting): authenticate will time out at 7879.879083s 2024-03-25T18:34:08.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.597+0000 7f632bfff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f632c14be90 0x7f632c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33072/0 (socket says 172.21.15.102:33072) 2024-03-25T18:34:08.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.597+0000 7f632bfff640 1 -- 172.21.15.102:0/3739388710 learned_addr learned my addr 172.21.15.102:0/3739388710 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:08.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.609+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156223626 0 0) 0x7f632c0af940 con 0x7f632c14be90 2024-03-25T18:34:08.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.609+0000 7f632a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.609+0000 7f632a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.609+0000 7f632a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.609+0000 7f632a7fc640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:34:08.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.609+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6314003520 con 0x7f632c14be90 2024-03-25T18:34:08.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.623+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301993585 0 0) 0x7f632c0b0a70 con 0x7f632c0b1730 2024-03-25T18:34:08.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.623+0000 7f632a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.623+0000 7f632a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.623+0000 7f632a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.623+0000 7f632a7fc640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:34:08.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.623+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c0af940 con 0x7f632c0b1730 2024-03-25T18:34:08.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494531553 0 0) 0x7f6314003520 con 0x7f632c14be90 2024-03-25T18:34:08.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 >> v1:172.21.15.102:6790/0 conn(0x7f632c0b4d60 legacy=0x7f632c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 >> v1:172.21.15.146:6789/0 conn(0x7f632c0b1730 legacy=0x7f632c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:08.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:34:08.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:08.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:34:08.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c0b1030 con 0x7f632c14be90 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6324001a10 con 0x7f632c14be90 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f6331fc6640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f6331fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f632a7fc640 10 monclient: dump: 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.634 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:08.635 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:08.635 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:08.635 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:08.635 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:08.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.632+0000 7f6331fc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:08.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.638+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6324001b70 con 0x7f632c14be90 2024-03-25T18:34:08.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.638+0000 7f632a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:08.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.638+0000 7f632a7fc640 1 -- 172.21.15.102:0/3739388710 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6324001f80 con 0x7f632c14be90 2024-03-25T18:34:08.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.638+0000 7f6331fc6640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:08.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.638+0000 7f6331fc6640 4 set_mon_vals no callback set 2024-03-25T18:34:08.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.638+0000 7f632a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.638+0000 7f632a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:08.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.638+0000 7f632a7fc640 10 monclient: dump: 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:08.640 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:08.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.639+0000 7f6331fc6640 10 monclient: shutdown 2024-03-25T18:34:08.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.639+0000 7f6331fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:08.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.639+0000 7f6331fc6640 1 -- 172.21.15.102:0/3739388710 >> v1:172.21.15.102:6789/0 conn(0x7f632c14be90 legacy=0x7f632c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.639+0000 7f6331fc6640 1 -- 172.21.15.102:0/3739388710 shutdown_connections 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.639+0000 7f6331fc6640 1 -- 172.21.15.102:0/3739388710 >> 172.21.15.102:0/3739388710 conn(0x7f632c0617a0 msgr2=0x7f632c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.639+0000 7f6331fc6640 1 -- 172.21.15.102:0/3739388710 shutdown_connections 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.639+0000 7f6331fc6640 1 -- 172.21.15.102:0/3739388710 wait complete. 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.640+0000 7f6331fc6640 10 monclient: build_initial_monmap 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.640+0000 7f6331fc6640 10 monclient: monmap: 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:34:08.642 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:08.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.640+0000 7f6331fc6640 1 librados: starting msgr at 2024-03-25T18:34:08.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.640+0000 7f6331fc6640 1 librados: starting objecter 2024-03-25T18:34:08.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.640+0000 7f6331fc6640 1 Processor -- start 2024-03-25T18:34:08.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.641+0000 7f6331fc6640 1 -- start start 2024-03-25T18:34:08.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.641+0000 7f6331fc6640 1 librados: setting wanted keys 2024-03-25T18:34:08.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.641+0000 7f6331fc6640 1 librados: calling monclient init 2024-03-25T18:34:08.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.641+0000 7f6331fc6640 10 monclient: init 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.641+0000 7f6331fc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.641+0000 7f6331fc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.642+0000 7f6331fc6640 10 monclient(hunting): picked mon.noname-c con 0x7f632c0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.642+0000 7f6331fc6640 10 monclient(hunting): picked mon.noname-a con 0x7f632c0b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.642+0000 7f6331fc6640 10 monclient(hunting): picked mon.noname-b con 0x7f632c14be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.642+0000 7f6331fc6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c1d86f0 con 0x7f632c0b4d60 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.642+0000 7f6331fc6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c13b4f0 con 0x7f632c14be90 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.642+0000 7f6331fc6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c1d7eb0 con 0x7f632c0b1730 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.642+0000 7f6331fc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:08.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.642+0000 7f6331fc6640 10 monclient(hunting): authenticate will time out at 7879.930117s 2024-03-25T18:34:08.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.648+0000 7f632affd640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f632c0b4d60 0x7f632c136fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33124/0 (socket says 172.21.15.102:33124) 2024-03-25T18:34:08.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.648+0000 7f632affd640 1 -- 172.21.15.102:0/3646298032 learned_addr learned my addr 172.21.15.102:0/3646298032 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:08.651 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.650+0000 7f6313fff640 10 client.?.objecter ms_handle_connect 0x7f632c0b4d60 2024-03-25T18:34:08.651 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.650+0000 7f6313fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.653+0000 7f6313fff640 10 client.?.objecter ms_handle_connect 0x7f632c0b1730 2024-03-25T18:34:08.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.653+0000 7f6313fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.659+0000 7f6313fff640 10 client.?.objecter ms_handle_connect 0x7f632c14be90 2024-03-25T18:34:08.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.659+0000 7f6313fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:08.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.667+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979456298 0 0) 0x7f632c13b4f0 con 0x7f632c14be90 2024-03-25T18:34:08.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.667+0000 7f6313fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.667+0000 7f6313fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.667+0000 7f6313fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.667+0000 7f6313fff640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:34:08.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.667+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6318004eb0 con 0x7f632c14be90 2024-03-25T18:34:08.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.668+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106642399 0 0) 0x7f632c0af940 con 0x7f632c0b4d60 2024-03-25T18:34:08.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.668+0000 7f6313fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.668+0000 7f6313fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.668+0000 7f6313fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.668+0000 7f6313fff640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:34:08.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.668+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c13b4f0 con 0x7f632c0b4d60 2024-03-25T18:34:08.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.681+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275230158 0 0) 0x7f632c1d7eb0 con 0x7f632c0b1730 2024-03-25T18:34:08.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.681+0000 7f6313fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:08.682 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.681+0000 7f6313fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:08.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.681+0000 7f6313fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:08.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.682+0000 7f6313fff640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:34:08.683 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.682+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c0af940 con 0x7f632c0b1730 2024-03-25T18:34:08.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.687+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3973084343 0 0) 0x7f632c13b4f0 con 0x7f632c0b4d60 2024-03-25T18:34:08.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.687+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f632c1d7eb0 con 0x7f632c0b4d60 2024-03-25T18:34:08.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.687+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6320001850 con 0x7f632c0b4d60 2024-03-25T18:34:08.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.687+0000 7f6313fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.687+0000 7f6313fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:08.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.687+0000 7f6313fff640 10 monclient(hunting): dump: 2024-03-25T18:34:08.688 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:08.688 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:08.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.687+0000 7f6313fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:08.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.693+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3591646495 0 0) 0x7f6318004eb0 con 0x7f632c14be90 2024-03-25T18:34:08.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.693+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f632c13b4f0 con 0x7f632c14be90 2024-03-25T18:34:08.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.693+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6324001e80 con 0x7f632c14be90 2024-03-25T18:34:08.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.693+0000 7f6313fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.693+0000 7f6313fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.693+0000 7f6313fff640 10 monclient(hunting): dump: 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:08.694 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:08.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2859407885 0 0) 0x7f632c1d7eb0 con 0x7f632c0b4d60 2024-03-25T18:34:08.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 >> v1:172.21.15.102:6790/0 conn(0x7f632c0b1730 legacy=0x7f632c1d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 >> v1:172.21.15.146:6789/0 conn(0x7f632c14be90 legacy=0x7f632c1e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:08.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:08.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 1 monclient: found mon.a 2024-03-25T18:34:08.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:08.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:34:08.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c1d8040 con 0x7f632c0b4d60 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 10 monclient: _finish_auth 0 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6313fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 10 monclient: _renew_subs 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c13ba70 con 0x7f632c0b4d60 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 10 monclient: _renew_subs 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c1378d0 con 0x7f632c0b4d60 2024-03-25T18:34:08.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.702+0000 7f6331fc6640 1 librados: init done 2024-03-25T18:34:08.705 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.703+0000 7f6331fc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:08.705 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.703+0000 7f6331fc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:08.705 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.703+0000 7f6331fc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:34:08.705 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.703+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c0627b0 con 0x7f632c0b4d60 2024-03-25T18:34:08.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.732+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6320001ce0 con 0x7f632c0b4d60 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.732+0000 7f6313fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.732+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f63200037b0 con 0x7f632c0b4d60 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.732+0000 7f6313fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.733+0000 7f6313fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.733+0000 7f6313fff640 10 monclient: dump: 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:08.734 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:08.735 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:08.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.733+0000 7f6329ffb640 4 set_mon_vals no callback set 2024-03-25T18:34:08.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.733+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f63200039d0 con 0x7f632c0b4d60 2024-03-25T18:34:08.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.734+0000 7f632b7fe640 10 monclient: get_auth_request con 0x7f631806df20 auth_method 0 2024-03-25T18:34:08.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:08.734+0000 7f6313fff640 10 client.4596.objecter ms_handle_connect 0x7f631806df20 2024-03-25T18:34:09.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.013+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f6320054700 con 0x7f632c0b4d60 2024-03-25T18:34:09.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.013+0000 7f6313fff640 10 client.4596.objecter ms_dispatch 0x7f632c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:09.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.013+0000 7f6313fff640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:09.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.013+0000 7f6313fff640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:09.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.013+0000 7f6313fff640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-25T18:34:09.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.013+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6320050210 con 0x7f632c0b4d60 2024-03-25T18:34:09.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.013+0000 7f6313fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:09.015 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.013+0000 7f6313fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:09.258 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.257+0000 7f6331fc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.258 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.257+0000 7f6331fc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.258 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.257+0000 7f6331fc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:34:09.258 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.257+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f632c069f40 con 0x7f632c0b4d60 2024-03-25T18:34:09.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.283+0000 7f6313fff640 1 -- 172.21.15.102:0/3646298032 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f6320052140 con 0x7f632c0b4d60 2024-03-25T18:34:09.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.283+0000 7f6313fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:09.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.283+0000 7f6313fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:09.284 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:34:09.284 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:09.287 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 10 librados: watch_flush enter 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 10 librados: watch_flush exit 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 10 client.4596.objecter successfully canceled tick 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 >> v1:172.21.15.146:6800/3848252714 conn(0x7f631806df20 legacy=0x7f63180703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 10 monclient: shutdown 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 >> v1:172.21.15.102:6789/0 conn(0x7f632c0b4d60 legacy=0x7f632c136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 shutdown_connections 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.286+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 >> 172.21.15.102:0/3646298032 conn(0x7f632c0617a0 msgr2=0x7f632c13c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.287+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 shutdown_connections 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.287+0000 7f6331fc6640 1 -- 172.21.15.102:0/3646298032 wait complete. 2024-03-25T18:34:09.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:09.287+0000 7f6331fc6640 1 librados: shutdown 2024-03-25T18:34:09.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:16.106 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.207+0000 7f2dd2b72640 10 monclient: get_monmap_and_config 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.207+0000 7f2dd2b72640 10 monclient: build_initial_monmap 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.207+0000 7f2dd2b72640 10 monclient: monmap: 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:34:16.208007+0000 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:34:16.208007+0000 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:34:16.208 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:34:16.209 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:16.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.208+0000 7f2dd2b72640 1 Processor -- start 2024-03-25T18:34:16.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.208+0000 7f2dd2b72640 1 -- start start 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.208+0000 7f2dd2b72640 10 monclient: init 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 10 monclient(hunting): picked mon.noname-a con 0x7f2dcc0b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 10 monclient(hunting): picked mon.noname-b con 0x7f2dcc0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 10 monclient(hunting): picked mon.noname-c con 0x7f2dcc14be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc0af940 con 0x7f2dcc0b4d60 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc0b0a70 con 0x7f2dcc14be90 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc0b0d50 con 0x7f2dcc0b1730 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:16.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.209+0000 7f2dd2b72640 10 monclient(hunting): authenticate will time out at 7887.497077s 2024-03-25T18:34:16.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.227+0000 7f2dcbfff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f2dcc0b1730 0x7f2dcc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:46776/0 (socket says 172.21.15.102:46776) 2024-03-25T18:34:16.228 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.227+0000 7f2dcbfff640 1 -- 172.21.15.102:0/2567980279 learned_addr learned my addr 172.21.15.102:0/2567980279 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:16.240 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.240+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049368690 0 0) 0x7f2dcc0b0d50 con 0x7f2dcc0b1730 2024-03-25T18:34:16.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.240+0000 7f2dcb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.240+0000 7f2dcb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.240+0000 7f2dcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.240+0000 7f2dcb7fe640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:34:16.241 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.240+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc003520 con 0x7f2dcc0b1730 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.254+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2579001245 0 0) 0x7f2dbc003520 con 0x7f2dcc0b1730 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.254+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 >> v1:172.21.15.146:6789/0 conn(0x7f2dcc14be90 legacy=0x7f2dcc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 >> v1:172.21.15.102:6789/0 conn(0x7f2dcc0b4d60 legacy=0x7f2dcc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 1 monclient: found mon.noname-b 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc0b1030 con 0x7f2dcc0b1730 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2db8001690 con 0x7f2dcc0b1730 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dcb7fe640 10 monclient: dump: 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dd2b72640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:34:16.257 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.255+0000 7f2dd2b72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.278+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db8001860 con 0x7f2dcc0b1730 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.278+0000 7f2dcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.278+0000 7f2dcb7fe640 1 -- 172.21.15.102:0/2567980279 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2db8001cb0 con 0x7f2dcc0b1730 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.278+0000 7f2dcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.278+0000 7f2dcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.278+0000 7f2dcb7fe640 10 monclient: dump: 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:16.279 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.278+0000 7f2dd2b72640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:16.280 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.278+0000 7f2dd2b72640 4 set_mon_vals no callback set 2024-03-25T18:34:16.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.279+0000 7f2dd2b72640 10 monclient: shutdown 2024-03-25T18:34:16.281 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.279+0000 7f2dd2b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.279+0000 7f2dd2b72640 1 -- 172.21.15.102:0/2567980279 >> v1:172.21.15.102:6790/0 conn(0x7f2dcc0b1730 legacy=0x7f2dcc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.279+0000 7f2dd2b72640 1 -- 172.21.15.102:0/2567980279 shutdown_connections 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.279+0000 7f2dd2b72640 1 -- 172.21.15.102:0/2567980279 >> 172.21.15.102:0/2567980279 conn(0x7f2dcc0617a0 msgr2=0x7f2dcc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.279+0000 7f2dd2b72640 1 -- 172.21.15.102:0/2567980279 shutdown_connections 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.279+0000 7f2dd2b72640 1 -- 172.21.15.102:0/2567980279 wait complete. 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.280+0000 7f2dd2b72640 10 monclient: build_initial_monmap 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.280+0000 7f2dd2b72640 10 monclient: monmap: 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:34:16.282 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.281+0000 7f2dd2b72640 1 librados: starting msgr at 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.281+0000 7f2dd2b72640 1 librados: starting objecter 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.281+0000 7f2dd2b72640 1 Processor -- start 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.281+0000 7f2dd2b72640 1 -- start start 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.281+0000 7f2dd2b72640 1 librados: setting wanted keys 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.281+0000 7f2dd2b72640 1 librados: calling monclient init 2024-03-25T18:34:16.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.281+0000 7f2dd2b72640 10 monclient: init 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 10 monclient(hunting): picked mon.noname-a con 0x7f2dcc0b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 10 monclient(hunting): picked mon.noname-c con 0x7f2dcc0b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 10 monclient(hunting): picked mon.noname-b con 0x7f2dcc14be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc13a280 con 0x7f2dcc0b1730 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc138d70 con 0x7f2dcc14be90 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dcc135710 con 0x7f2dcc0b4d60 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:16.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.282+0000 7f2dd2b72640 10 monclient(hunting): authenticate will time out at 7887.570437s 2024-03-25T18:34:16.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.291+0000 7f2dd110f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2dcc14be90 0x7f2dcc1db0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:58826/0 (socket says 172.21.15.102:58826) 2024-03-25T18:34:16.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.291+0000 7f2dd110f640 1 -- 172.21.15.102:0/3801489489 learned_addr learned my addr 172.21.15.102:0/3801489489 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:16.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.300+0000 7f2dc97fa640 10 client.?.objecter ms_handle_connect 0x7f2dcc14be90 2024-03-25T18:34:16.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.300+0000 7f2dc97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.300+0000 7f2dc97fa640 10 client.?.objecter ms_handle_connect 0x7f2dcc0b4d60 2024-03-25T18:34:16.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.300+0000 7f2dc97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.308 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.308+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268303199 0 0) 0x7f2dcc138d70 con 0x7f2dcc14be90 2024-03-25T18:34:16.308 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.308+0000 7f2dc97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.308+0000 7f2dc97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.308+0000 7f2dc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.308+0000 7f2dc97fa640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:34:16.309 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.308+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0004eb0 con 0x7f2dcc14be90 2024-03-25T18:34:16.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.314+0000 7f2dc97fa640 10 client.?.objecter ms_handle_connect 0x7f2dcc0b1730 2024-03-25T18:34:16.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.314+0000 7f2dc97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:16.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.314+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429033992 0 0) 0x7f2dcc135710 con 0x7f2dcc0b4d60 2024-03-25T18:34:16.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.314+0000 7f2dc97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.314+0000 7f2dc97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.315 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.314+0000 7f2dc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.314+0000 7f2dc97fa640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:34:16.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.314+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dcc138d70 con 0x7f2dcc0b4d60 2024-03-25T18:34:16.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.320+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3644222237 0 0) 0x7f2db0004eb0 con 0x7f2dcc14be90 2024-03-25T18:34:16.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.320+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dcc135710 con 0x7f2dcc14be90 2024-03-25T18:34:16.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.320+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2dc4001ad0 con 0x7f2dcc14be90 2024-03-25T18:34:16.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.320+0000 7f2dc97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.320+0000 7f2dc97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:16.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.320+0000 7f2dc97fa640 10 monclient(hunting): dump: 2024-03-25T18:34:16.321 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:16.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.320+0000 7f2dc97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:16.326 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.325+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133296598 0 0) 0x7f2dcc13a280 con 0x7f2dcc0b1730 2024-03-25T18:34:16.326 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.325+0000 7f2dc97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:34:16.326 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.325+0000 7f2dc97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:16.326 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.325+0000 7f2dc97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:16.326 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.325+0000 7f2dc97fa640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:34:16.326 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.325+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0004eb0 con 0x7f2dcc0b1730 2024-03-25T18:34:16.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780107899 0 0) 0x7f2dcc135710 con 0x7f2dcc14be90 2024-03-25T18:34:16.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 >> v1:172.21.15.102:6790/0 conn(0x7f2dcc0b4d60 legacy=0x7f2dcc1da9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 >> v1:172.21.15.102:6789/0 conn(0x7f2dcc0b1730 legacy=0x7f2dcc135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 1 monclient: found mon.b 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc1358a0 con 0x7f2dcc14be90 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 10 monclient: _finish_auth 0 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 10 monclient: _renew_subs 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc142b00 con 0x7f2dcc14be90 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:16.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 10 monclient: _renew_subs 2024-03-25T18:34:16.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:16.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc1384f0 con 0x7f2dcc14be90 2024-03-25T18:34:16.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.332+0000 7f2dd2b72640 1 librados: init done 2024-03-25T18:34:16.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.333+0000 7f2dd2b72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.333+0000 7f2dd2b72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.333+0000 7f2dd2b72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:16.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.333+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc0627b0 con 0x7f2dcc14be90 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.602+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc4001fe0 con 0x7f2dcc14be90 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.602+0000 7f2dc97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.602+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2dc4004d80 con 0x7f2dcc14be90 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.602+0000 7f2dc97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.602+0000 7f2dc97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.602+0000 7f2dc97fa640 10 monclient: dump: 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:16.603 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:16.604 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:16.604 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:16.604 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:16.604 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:16.604 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:16.604 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:16.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.602+0000 7f2dcaffd640 4 set_mon_vals no callback set 2024-03-25T18:34:16.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.603+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f2dc401ad50 con 0x7f2dcc14be90 2024-03-25T18:34:16.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.603+0000 7f2dd090e640 10 monclient: get_auth_request con 0x7f2db006ddb0 auth_method 0 2024-03-25T18:34:16.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.603+0000 7f2dc97fa640 10 client.4603.objecter ms_handle_connect 0x7f2db006ddb0 2024-03-25T18:34:16.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.609+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f2dc40549a0 con 0x7f2dcc14be90 2024-03-25T18:34:16.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.609+0000 7f2dc97fa640 10 client.4603.objecter ms_dispatch 0x7f2dcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:16.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.609+0000 7f2dc97fa640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:16.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.609+0000 7f2dc97fa640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:16.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.609+0000 7f2dc97fa640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T18:34:16.610 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.610+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dc40502e0 con 0x7f2dcc14be90 2024-03-25T18:34:16.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.610+0000 7f2dc97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:16.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.610+0000 7f2dc97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:16.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.860+0000 7f2dd2b72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.860+0000 7f2dd2b72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.860+0000 7f2dd2b72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:16.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.860+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dcc138f00 con 0x7f2dcc14be90 2024-03-25T18:34:16.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.895+0000 7f2dc97fa640 1 -- 172.21.15.102:0/3801489489 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f2dc40523e0 con 0x7f2dcc14be90 2024-03-25T18:34:16.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.896+0000 7f2dc97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:16.896 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.896+0000 7f2dc97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:16.897 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:34:16.897 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.897+0000 7f2dd2b72640 10 librados: watch_flush enter 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.897+0000 7f2dd2b72640 10 librados: watch_flush exit 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.897+0000 7f2dd2b72640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.897+0000 7f2dd2b72640 10 client.4603.objecter successfully canceled tick 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 >> v1:172.21.15.146:6800/3848252714 conn(0x7f2db006ddb0 legacy=0x7f2db0070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 10 monclient: shutdown 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 >> v1:172.21.15.146:6789/0 conn(0x7f2dcc14be90 legacy=0x7f2dcc1db0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 shutdown_connections 2024-03-25T18:34:16.899 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 >> 172.21.15.102:0/3801489489 conn(0x7f2dcc0617a0 msgr2=0x7f2dcc0b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:16.900 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 shutdown_connections 2024-03-25T18:34:16.900 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 1 -- 172.21.15.102:0/3801489489 wait complete. 2024-03-25T18:34:16.900 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:16.898+0000 7f2dd2b72640 1 librados: shutdown 2024-03-25T18:34:16.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:23.717 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:23.830 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.829+0000 7f6d977cf640 10 monclient: get_monmap_and_config 2024-03-25T18:34:23.830 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.829+0000 7f6d977cf640 10 monclient: build_initial_monmap 2024-03-25T18:34:23.830 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.829+0000 7f6d977cf640 10 monclient: monmap: 2024-03-25T18:34:23.830 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:23.830 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:23.830 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:34:23.830254+0000 2024-03-25T18:34:23.830 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:34:23.830254+0000 2024-03-25T18:34:23.830 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:23.831 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:23.831 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:23.831 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:34:23.831 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:34:23.831 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:23.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.830+0000 7f6d977cf640 1 Processor -- start 2024-03-25T18:34:23.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.830+0000 7f6d977cf640 1 -- start start 2024-03-25T18:34:23.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.830+0000 7f6d977cf640 10 monclient: init 2024-03-25T18:34:23.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.830+0000 7f6d977cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:23.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.830+0000 7f6d977cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:34:23.832 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.831+0000 7f6d977cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6d900b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.831+0000 7f6d977cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6d900b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.831+0000 7f6d977cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9014be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.831+0000 7f6d977cf640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d900af940 con 0x7f6d900b4d60 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.831+0000 7f6d977cf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d900b0a70 con 0x7f6d9014be90 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.831+0000 7f6d977cf640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d900b0d50 con 0x7f6d900b1730 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.831+0000 7f6d977cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.831+0000 7f6d977cf640 10 monclient(hunting): authenticate will time out at 7895.119027s 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.832+0000 7f6d94d6a640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f6d900b1730 0x7f6d900b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:37708/0 (socket says 172.21.15.102:37708) 2024-03-25T18:34:23.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.832+0000 7f6d94d6a640 1 -- 172.21.15.102:0/1998118585 learned_addr learned my addr 172.21.15.102:0/1998118585 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:23.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.844+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047671334 0 0) 0x7f6d900b0d50 con 0x7f6d900b1730 2024-03-25T18:34:23.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.844+0000 7f6d87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.844+0000 7f6d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.844+0000 7f6d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.844+0000 7f6d87fff640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:34:23.845 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.844+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c003520 con 0x7f6d900b1730 2024-03-25T18:34:23.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 662855817 0 0) 0x7f6d7c003520 con 0x7f6d900b1730 2024-03-25T18:34:23.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 >> v1:172.21.15.146:6789/0 conn(0x7f6d9014be90 legacy=0x7f6d9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 >> v1:172.21.15.102:6789/0 conn(0x7f6d900b4d60 legacy=0x7f6d9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:23.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 1 monclient: found mon.noname-b 2024-03-25T18:34:23.857 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d900b1030 con 0x7f6d900b1730 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 10 monclient: _finish_auth 0 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d80001690 con 0x7f6d900b1730 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d87fff640 10 monclient: dump: 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.858 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d977cf640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:34:23.859 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.856+0000 7f6d977cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:23.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.864+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d800017f0 con 0x7f6d900b1730 2024-03-25T18:34:23.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.864+0000 7f6d87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:23.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.864+0000 7f6d87fff640 1 -- 172.21.15.102:0/1998118585 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d80001c00 con 0x7f6d900b1730 2024-03-25T18:34:23.865 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.864+0000 7f6d977cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.864+0000 7f6d977cf640 4 set_mon_vals no callback set 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.864+0000 7f6d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.864+0000 7f6d87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.864+0000 7f6d87fff640 10 monclient: dump: 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:23.866 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.865+0000 7f6d977cf640 10 monclient: shutdown 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.865+0000 7f6d977cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.865+0000 7f6d977cf640 1 -- 172.21.15.102:0/1998118585 >> v1:172.21.15.102:6790/0 conn(0x7f6d900b1730 legacy=0x7f6d900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.865+0000 7f6d977cf640 1 -- 172.21.15.102:0/1998118585 shutdown_connections 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.865+0000 7f6d977cf640 1 -- 172.21.15.102:0/1998118585 >> 172.21.15.102:0/1998118585 conn(0x7f6d900617a0 msgr2=0x7f6d90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.865+0000 7f6d977cf640 1 -- 172.21.15.102:0/1998118585 shutdown_connections 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.865+0000 7f6d977cf640 1 -- 172.21.15.102:0/1998118585 wait complete. 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.866+0000 7f6d977cf640 10 monclient: build_initial_monmap 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.866+0000 7f6d977cf640 10 monclient: monmap: 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:34:23.869 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.866+0000 7f6d977cf640 1 librados: starting msgr at 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.866+0000 7f6d977cf640 1 librados: starting objecter 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.866+0000 7f6d977cf640 1 Processor -- start 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 1 -- start start 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 1 librados: setting wanted keys 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 1 librados: calling monclient init 2024-03-25T18:34:23.870 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 10 monclient: init 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 10 monclient(hunting): picked mon.noname-c con 0x7f6d900b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 10 monclient(hunting): picked mon.noname-a con 0x7f6d900b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9014be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9013b810 con 0x7f6d900b4d60 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9013a730 con 0x7f6d9014be90 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d90139140 con 0x7f6d900b1730 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:23.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.867+0000 7f6d977cf640 10 monclient(hunting): authenticate will time out at 7895.155900s 2024-03-25T18:34:23.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.872+0000 7f6d94d6a640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6d900b4d60 0x7f6d9012fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46974/0 (socket says 172.21.15.102:46974) 2024-03-25T18:34:23.873 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.872+0000 7f6d94d6a640 1 -- 172.21.15.102:0/4157513142 learned_addr learned my addr 172.21.15.102:0/4157513142 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:23.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.874+0000 7f6d85ffb640 10 client.?.objecter ms_handle_connect 0x7f6d900b4d60 2024-03-25T18:34:23.875 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.874+0000 7f6d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.879+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650985794 0 0) 0x7f6d9013b810 con 0x7f6d900b4d60 2024-03-25T18:34:23.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.879+0000 7f6d85ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.879+0000 7f6d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.879+0000 7f6d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.879+0000 7f6d85ffb640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:34:23.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.879+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d74004eb0 con 0x7f6d900b4d60 2024-03-25T18:34:23.889 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.888+0000 7f6d85ffb640 10 client.?.objecter ms_handle_connect 0x7f6d9014be90 2024-03-25T18:34:23.889 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.888+0000 7f6d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.890 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.890+0000 7f6d85ffb640 10 client.?.objecter ms_handle_connect 0x7f6d900b1730 2024-03-25T18:34:23.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.890+0000 7f6d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:23.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.892+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521666138 0 0) 0x7f6d9013a730 con 0x7f6d9014be90 2024-03-25T18:34:23.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.892+0000 7f6d85ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.892+0000 7f6d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.892+0000 7f6d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.892+0000 7f6d85ffb640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:34:23.893 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.892+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9013b810 con 0x7f6d9014be90 2024-03-25T18:34:23.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.900+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367295306 0 0) 0x7f6d90139140 con 0x7f6d900b1730 2024-03-25T18:34:23.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.900+0000 7f6d85ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:23.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.900+0000 7f6d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:23.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.900+0000 7f6d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:23.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.900+0000 7f6d85ffb640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:34:23.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.900+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9013a730 con 0x7f6d900b1730 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.914+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 551534233 0 0) 0x7f6d74004eb0 con 0x7f6d900b4d60 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.914+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d90139140 con 0x7f6d900b4d60 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.914+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d80001c00 con 0x7f6d900b4d60 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.914+0000 7f6d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.914+0000 7f6d85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.914+0000 7f6d85ffb640 10 monclient(hunting): dump: 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.915 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.916 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:23.916 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:23.916 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:23.916 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:23.916 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:23.916 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.914+0000 7f6d85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:23.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3416277835 0 0) 0x7f6d9013a730 con 0x7f6d900b1730 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d74004eb0 con 0x7f6d900b1730 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d8c001830 con 0x7f6d900b1730 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 10 monclient(hunting): dump: 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:23.919 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3832088514 0 0) 0x7f6d9013b810 con 0x7f6d9014be90 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9013a730 con 0x7f6d9014be90 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d88001b00 con 0x7f6d9014be90 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.918+0000 7f6d85ffb640 10 monclient(hunting): dump: 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:23.920 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:23.921 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:23.921 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:23.921 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:23.921 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:23.921 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:23.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.924+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1477889269 0 0) 0x7f6d9013a730 con 0x7f6d9014be90 2024-03-25T18:34:23.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 >> v1:172.21.15.102:6790/0 conn(0x7f6d900b1730 legacy=0x7f6d90139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 >> v1:172.21.15.102:6789/0 conn(0x7f6d900b4d60 legacy=0x7f6d9012fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:23.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:34:23.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 1 monclient: found mon.b 2024-03-25T18:34:23.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:23.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:23.926 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9013bbc0 con 0x7f6d9014be90 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 10 monclient: _finish_auth 0 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 5 monclient: authenticate success, global_id 4609 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 10 monclient: _renew_subs 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9013a330 con 0x7f6d9014be90 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 10 monclient: _renew_subs 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d901307e0 con 0x7f6d9014be90 2024-03-25T18:34:23.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.925+0000 7f6d977cf640 1 librados: init done 2024-03-25T18:34:23.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.926+0000 7f6d977cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:23.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.926+0000 7f6d977cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:23.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.926+0000 7f6d977cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:23.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:23.926+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d900627b0 con 0x7f6d9014be90 2024-03-25T18:34:24.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.208+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d88001fd0 con 0x7f6d9014be90 2024-03-25T18:34:24.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.208+0000 7f6d85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:24.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.208+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f6d88003b10 con 0x7f6d9014be90 2024-03-25T18:34:24.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.208+0000 7f6d85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:24.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.208+0000 7f6d85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:24.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.208+0000 7f6d85ffb640 10 monclient: dump: 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:24.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.208+0000 7f6d877fe640 4 set_mon_vals no callback set 2024-03-25T18:34:24.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.209+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f6d88019ae0 con 0x7f6d9014be90 2024-03-25T18:34:24.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.210+0000 7f6d9556b640 10 monclient: get_auth_request con 0x7f6d7406e110 auth_method 0 2024-03-25T18:34:24.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.210+0000 7f6d85ffb640 10 client.4609.objecter ms_handle_connect 0x7f6d7406e110 2024-03-25T18:34:24.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.215+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f6d880536f0 con 0x7f6d9014be90 2024-03-25T18:34:24.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.215+0000 7f6d85ffb640 10 client.4609.objecter ms_dispatch 0x7f6d90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:24.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.215+0000 7f6d85ffb640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:24.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.215+0000 7f6d85ffb640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:24.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.215+0000 7f6d85ffb640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T18:34:24.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.217+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d8804f030 con 0x7f6d9014be90 2024-03-25T18:34:24.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.217+0000 7f6d85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:24.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.217+0000 7f6d85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:24.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.464+0000 7f6d977cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.464+0000 7f6d977cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.464+0000 7f6d977cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:24.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.464+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d9013bd50 con 0x7f6d9014be90 2024-03-25T18:34:24.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.583+0000 7f6d85ffb640 1 -- 172.21.15.102:0/4157513142 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f6d88051130 con 0x7f6d9014be90 2024-03-25T18:34:24.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.583+0000 7f6d85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:24.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.583+0000 7f6d85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:24.584 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:34:24.585 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:24.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.585+0000 7f6d977cf640 10 librados: watch_flush enter 2024-03-25T18:34:24.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.585+0000 7f6d977cf640 10 librados: watch_flush exit 2024-03-25T18:34:24.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.585+0000 7f6d977cf640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T18:34:24.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.585+0000 7f6d977cf640 10 client.4609.objecter successfully canceled tick 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 >> v1:172.21.15.146:6800/3848252714 conn(0x7f6d7406e110 legacy=0x7f6d740705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 10 monclient: shutdown 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 >> v1:172.21.15.146:6789/0 conn(0x7f6d9014be90 legacy=0x7f6d901d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 shutdown_connections 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 >> 172.21.15.102:0/4157513142 conn(0x7f6d900617a0 msgr2=0x7f6d90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 shutdown_connections 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 1 -- 172.21.15.102:0/4157513142 wait complete. 2024-03-25T18:34:24.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:24.586+0000 7f6d977cf640 1 librados: shutdown 2024-03-25T18:34:24.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:29.377 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:45.612 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:29.427 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:45.663 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:29.448 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:08:45.684 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:29.476 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:45.711 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:29.479 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:45.692 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:29.483 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:45.719 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:29.497 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:45.710 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:29.515 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:08:45.751 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:29.535 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:08:45.748 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:31.405 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:31.517 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.516+0000 7f65b9a3e640 10 monclient: get_monmap_and_config 2024-03-25T18:34:31.517 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.516+0000 7f65b9a3e640 10 monclient: build_initial_monmap 2024-03-25T18:34:31.517 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.516+0000 7f65b9a3e640 10 monclient: monmap: 2024-03-25T18:34:31.517 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:31.517 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:31.517 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:34:31.517371+0000 2024-03-25T18:34:31.518 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:34:31.517371+0000 2024-03-25T18:34:31.518 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:31.518 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:31.518 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:31.518 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:34:31.518 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:34:31.518 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:31.519 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.517+0000 7f65b9a3e640 1 Processor -- start 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.517+0000 7f65b9a3e640 1 -- start start 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.517+0000 7f65b9a3e640 10 monclient: init 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f65b40a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f65b413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f65b41430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b4142520 con 0x7f65b413f710 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40b28a0 con 0x7f65b41430c0 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40b2a30 con 0x7f65b40a98f0 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:31.520 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.518+0000 7f65b9a3e640 10 monclient(hunting): authenticate will time out at 7902.806377s 2024-03-25T18:34:31.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.523+0000 7f65b2ffd640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f65b40a98f0 0x7f65b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:55950/0 (socket says 172.21.15.102:55950) 2024-03-25T18:34:31.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.523+0000 7f65b2ffd640 1 -- 172.21.15.102:0/4177724261 learned_addr learned my addr 172.21.15.102:0/4177724261 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:31.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.535+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015894321 0 0) 0x7f65b40b2a30 con 0x7f65b40a98f0 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.535+0000 7f65b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.535+0000 7f65b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.535+0000 7f65b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.535+0000 7f65b27fc640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.535+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a0003520 con 0x7f65b40a98f0 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.536+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138138237 0 0) 0x7f65b40b28a0 con 0x7f65b41430c0 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.536+0000 7f65b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.536+0000 7f65b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.536+0000 7f65b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.536+0000 7f65b27fc640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:34:31.537 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.536+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b40b2a30 con 0x7f65b41430c0 2024-03-25T18:34:31.545 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.544+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872421659 0 0) 0x7f65b4142520 con 0x7f65b413f710 2024-03-25T18:34:31.545 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.544+0000 7f65b27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.545 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.544+0000 7f65b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.545 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.544+0000 7f65b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.545 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.544+0000 7f65b27fc640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:34:31.545 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.544+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b40b28a0 con 0x7f65b413f710 2024-03-25T18:34:31.546 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.544+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012295456 0 0) 0x7f65b40b2a30 con 0x7f65b41430c0 2024-03-25T18:34:31.546 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 >> v1:172.21.15.102:6790/0 conn(0x7f65b40a98f0 legacy=0x7f65b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 >> v1:172.21.15.102:6789/0 conn(0x7f65b413f710 legacy=0x7f65b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 1 monclient: found mon.noname-c 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b40b44c0 con 0x7f65b41430c0 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f65ac0019f0 con 0x7f65b41430c0 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.547 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b27fc640 10 monclient: dump: 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:31.548 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b9a3e640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:34:31.549 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.545+0000 7f65b9a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:31.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.557+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65ac001b50 con 0x7f65b41430c0 2024-03-25T18:34:31.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.557+0000 7f65b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:31.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.557+0000 7f65b27fc640 1 -- 172.21.15.102:0/4177724261 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f65ac001f60 con 0x7f65b41430c0 2024-03-25T18:34:31.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.557+0000 7f65b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.559 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.557+0000 7f65b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.557+0000 7f65b27fc640 10 monclient: dump: 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:31.560 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 4 set_mon_vals no callback set 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 10 monclient: shutdown 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 1 -- 172.21.15.102:0/4177724261 >> v1:172.21.15.146:6789/0 conn(0x7f65b41430c0 legacy=0x7f65b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 1 -- 172.21.15.102:0/4177724261 shutdown_connections 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 1 -- 172.21.15.102:0/4177724261 >> 172.21.15.102:0/4177724261 conn(0x7f65b40b7ea0 msgr2=0x7f65b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 1 -- 172.21.15.102:0/4177724261 shutdown_connections 2024-03-25T18:34:31.561 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.558+0000 7f65b9a3e640 1 -- 172.21.15.102:0/4177724261 wait complete. 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 10 monclient: build_initial_monmap 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 10 monclient: monmap: 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:34:31.563 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 1 librados: starting msgr at 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 1 librados: starting objecter 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 1 Processor -- start 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 1 -- start start 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 1 librados: setting wanted keys 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 1 librados: calling monclient init 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.559+0000 7f65b9a3e640 10 monclient: init 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 10 monclient(hunting): picked mon.noname-c con 0x7f65b40a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:31.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 10 monclient(hunting): picked mon.noname-b con 0x7f65b413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:31.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 10 monclient(hunting): picked mon.noname-a con 0x7f65b41430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:31.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b4139090 con 0x7f65b41430c0 2024-03-25T18:34:31.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b41db010 con 0x7f65b413f710 2024-03-25T18:34:31.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b41dac50 con 0x7f65b40a98f0 2024-03-25T18:34:31.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:31.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.560+0000 7f65b9a3e640 10 monclient(hunting): authenticate will time out at 7902.848517s 2024-03-25T18:34:31.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.566+0000 7f65abfff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f65b413f710 0x7f65b41db530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:57164/0 (socket says 172.21.15.102:57164) 2024-03-25T18:34:31.567 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.566+0000 7f65abfff640 1 -- 172.21.15.102:0/1738515490 learned_addr learned my addr 172.21.15.102:0/1738515490 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:31.568 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.567+0000 7f65b0ff9640 10 client.?.objecter ms_handle_connect 0x7f65b413f710 2024-03-25T18:34:31.568 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.567+0000 7f65b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.569+0000 7f65b0ff9640 10 client.?.objecter ms_handle_connect 0x7f65b40a98f0 2024-03-25T18:34:31.570 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.569+0000 7f65b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.576+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229653381 0 0) 0x7f65b41db010 con 0x7f65b413f710 2024-03-25T18:34:31.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.576+0000 7f65b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.576+0000 7f65b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.576+0000 7f65b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.576+0000 7f65b0ff9640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:34:31.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.576+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6594004eb0 con 0x7f65b413f710 2024-03-25T18:34:31.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.577+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443589474 0 0) 0x7f65b41dac50 con 0x7f65b40a98f0 2024-03-25T18:34:31.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.577+0000 7f65b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.577+0000 7f65b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.577+0000 7f65b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.577+0000 7f65b0ff9640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:34:31.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.577+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b41db010 con 0x7f65b40a98f0 2024-03-25T18:34:31.582 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.581+0000 7f65b0ff9640 10 client.?.objecter ms_handle_connect 0x7f65b41430c0 2024-03-25T18:34:31.582 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.581+0000 7f65b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:31.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.586+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857628805 0 0) 0x7f6594004eb0 con 0x7f65b413f710 2024-03-25T18:34:31.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.586+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65b41dac50 con 0x7f65b413f710 2024-03-25T18:34:31.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.586+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f659c001aa0 con 0x7f65b413f710 2024-03-25T18:34:31.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.586+0000 7f65b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.586+0000 7f65b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:34:31.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.586+0000 7f65b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:31.588 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.586+0000 7f65b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.596+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2079062311 0 0) 0x7f65b41db010 con 0x7f65b40a98f0 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.596+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6594004eb0 con 0x7f65b40a98f0 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.596+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f65a4001950 con 0x7f65b40a98f0 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.596+0000 7f65b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.596+0000 7f65b0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.596+0000 7f65b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.597 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.598 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:31.598 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:31.598 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:31.598 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:31.598 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:31.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.597+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510393615 0 0) 0x7f65b4139090 con 0x7f65b41430c0 2024-03-25T18:34:31.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.597+0000 7f65b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:34:31.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.597+0000 7f65b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:31.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.597+0000 7f65b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:31.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.597+0000 7f65b0ff9640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:34:31.600 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.597+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b41db010 con 0x7f65b41430c0 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3591001933 0 0) 0x7f65b41dac50 con 0x7f65b413f710 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 >> v1:172.21.15.102:6790/0 conn(0x7f65b40a98f0 legacy=0x7f65b4139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 >> v1:172.21.15.102:6789/0 conn(0x7f65b41430c0 legacy=0x7f65b41e21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 1 monclient: found mon.b 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b41dade0 con 0x7f65b413f710 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:34:31.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 10 monclient: _renew_subs 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b40b7240 con 0x7f65b413f710 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 10 client.4615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 10 monclient: _renew_subs 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b4134f90 con 0x7f65b413f710 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.598+0000 7f65b9a3e640 1 librados: init done 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.599+0000 7f65b9a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.599+0000 7f65b9a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.599+0000 7f65b9a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:31.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.599+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b414f2d0 con 0x7f65b413f710 2024-03-25T18:34:31.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.879+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f659c001d00 con 0x7f65b413f710 2024-03-25T18:34:31.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.879+0000 7f65b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:31.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.879+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f659c003a60 con 0x7f65b413f710 2024-03-25T18:34:31.880 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.879+0000 7f65b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.879+0000 7f65b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.879+0000 7f65b0ff9640 10 monclient: dump: 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:31.881 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:31.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.879+0000 7f65b1ffb640 4 set_mon_vals no callback set 2024-03-25T18:34:31.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.879+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f659c003c80 con 0x7f65b413f710 2024-03-25T18:34:31.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.880+0000 7f65b2ffd640 10 monclient: get_auth_request con 0x7f659406df00 auth_method 0 2024-03-25T18:34:31.882 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.880+0000 7f65b0ff9640 10 client.4615.objecter ms_handle_connect 0x7f659406df00 2024-03-25T18:34:31.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.884+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f659c054960 con 0x7f65b413f710 2024-03-25T18:34:31.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.884+0000 7f65b0ff9640 10 client.4615.objecter ms_dispatch 0x7f65b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:31.885 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.884+0000 7f65b0ff9640 3 client.4615.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:31.886 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.884+0000 7f65b0ff9640 3 client.4615.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:31.886 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.884+0000 7f65b0ff9640 20 client.4615.objecter dump_active .. 0 homeless 2024-03-25T18:34:31.886 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.884+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f659c0502a0 con 0x7f65b413f710 2024-03-25T18:34:31.886 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.884+0000 7f65b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:31.886 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:31.884+0000 7f65b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:32.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.121+0000 7f65b9a3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:32.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.121+0000 7f65b9a3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:32.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.121+0000 7f65b9a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:32.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.121+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65b41db1a0 con 0x7f65b413f710 2024-03-25T18:34:32.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.206+0000 7f65b0ff9640 1 -- 172.21.15.102:0/1738515490 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f659c0523a0 con 0x7f65b413f710 2024-03-25T18:34:32.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.206+0000 7f65b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:32.208 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.206+0000 7f65b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:32.208 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:34:32.208 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.208+0000 7f65b9a3e640 10 librados: watch_flush enter 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.208+0000 7f65b9a3e640 10 librados: watch_flush exit 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.208+0000 7f65b9a3e640 20 client.4615.objecter shutdown clearing up homeless session... 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.208+0000 7f65b9a3e640 10 client.4615.objecter successfully canceled tick 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 >> v1:172.21.15.146:6800/3848252714 conn(0x7f659406df00 legacy=0x7f65940703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 10 monclient: shutdown 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 >> v1:172.21.15.146:6789/0 conn(0x7f65b413f710 legacy=0x7f65b41db530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:32.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 shutdown_connections 2024-03-25T18:34:32.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 >> 172.21.15.102:0/1738515490 conn(0x7f65b40b7ea0 msgr2=0x7f65b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:32.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 shutdown_connections 2024-03-25T18:34:32.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 1 -- 172.21.15.102:0/1738515490 wait complete. 2024-03-25T18:34:32.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:32.209+0000 7f65b9a3e640 1 librados: shutdown 2024-03-25T18:34:32.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:39.027 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.142+0000 7f62c1e8d640 10 monclient: get_monmap_and_config 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.142+0000 7f62c1e8d640 10 monclient: build_initial_monmap 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.142+0000 7f62c1e8d640 10 monclient: monmap: 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:34:39.142870+0000 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:34:39.142870+0000 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:39.143 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:39.144 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:39.144 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:34:39.144 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:34:39.144 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:39.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.143+0000 7f62c1e8d640 1 Processor -- start 2024-03-25T18:34:39.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.143+0000 7f62c1e8d640 1 -- start start 2024-03-25T18:34:39.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.143+0000 7f62c1e8d640 10 monclient: init 2024-03-25T18:34:39.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.143+0000 7f62c1e8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:39.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:39.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f62bc0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:39.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f62bc13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:39.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f62bc1430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:39.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc142520 con 0x7f62bc1430c0 2024-03-25T18:34:39.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc0b28a0 con 0x7f62bc0a98f0 2024-03-25T18:34:39.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc0b2a30 con 0x7f62bc13f710 2024-03-25T18:34:39.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:39.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.144+0000 7f62c1e8d640 10 monclient(hunting): authenticate will time out at 7910.432166s 2024-03-25T18:34:39.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.151+0000 7f62bb7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f62bc0a98f0 0x7f62bc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:33632/0 (socket says 172.21.15.102:33632) 2024-03-25T18:34:39.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.151+0000 7f62bb7fe640 1 -- 172.21.15.102:0/1735469416 learned_addr learned my addr 172.21.15.102:0/1735469416 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:39.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.161+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809464857 0 0) 0x7f62bc0b28a0 con 0x7f62bc0a98f0 2024-03-25T18:34:39.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.161+0000 7f62ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.161+0000 7f62ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.161+0000 7f62ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.161 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.161+0000 7f62ba7fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:34:39.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.161+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a4003520 con 0x7f62bc0a98f0 2024-03-25T18:34:39.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.173+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786714404 0 0) 0x7f62bc142520 con 0x7f62bc1430c0 2024-03-25T18:34:39.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.173+0000 7f62ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.173+0000 7f62ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.173+0000 7f62ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.173+0000 7f62ba7fc640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:34:39.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.173+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc0b28a0 con 0x7f62bc1430c0 2024-03-25T18:34:39.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.175+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477557981 0 0) 0x7f62bc0b2a30 con 0x7f62bc13f710 2024-03-25T18:34:39.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.175+0000 7f62ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.175+0000 7f62ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.175+0000 7f62ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.175+0000 7f62ba7fc640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:34:39.176 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.175+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc142520 con 0x7f62bc13f710 2024-03-25T18:34:39.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 807600100 0 0) 0x7f62bc0b28a0 con 0x7f62bc1430c0 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 >> v1:172.21.15.102:6790/0 conn(0x7f62bc13f710 legacy=0x7f62bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 >> v1:172.21.15.146:6789/0 conn(0x7f62bc0a98f0 legacy=0x7f62bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 1 monclient: found mon.noname-a 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc0b44c0 con 0x7f62bc1430c0 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62c1e8d640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:34:39.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.181+0000 7f62c1e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:39.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f62b40019b0 con 0x7f62bc1430c0 2024-03-25T18:34:39.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.200 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:34:39.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 10 monclient: dump: 2024-03-25T18:34:39.201 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:39.201 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:39.201 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.201 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.201 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:39.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62b4001c00 con 0x7f62bc1430c0 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 1 -- 172.21.15.102:0/1735469416 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f62b4001f90 con 0x7f62bc1430c0 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62c1e8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62c1e8d640 4 set_mon_vals no callback set 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.203 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.199+0000 7f62ba7fc640 10 monclient: dump: 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.204 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:39.205 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:39.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.200+0000 7f62c1e8d640 10 monclient: shutdown 2024-03-25T18:34:39.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.200+0000 7f62c1e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:39.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.200+0000 7f62c1e8d640 1 -- 172.21.15.102:0/1735469416 >> v1:172.21.15.102:6789/0 conn(0x7f62bc1430c0 legacy=0x7f62bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.200+0000 7f62c1e8d640 1 -- 172.21.15.102:0/1735469416 shutdown_connections 2024-03-25T18:34:39.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.200+0000 7f62c1e8d640 1 -- 172.21.15.102:0/1735469416 >> 172.21.15.102:0/1735469416 conn(0x7f62bc0b7ea0 msgr2=0x7f62bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:39.209 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.200+0000 7f62c1e8d640 1 -- 172.21.15.102:0/1735469416 shutdown_connections 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.200+0000 7f62c1e8d640 1 -- 172.21.15.102:0/1735469416 wait complete. 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 10 monclient: build_initial_monmap 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 10 monclient: monmap: 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 1 librados: starting msgr at 2024-03-25T18:34:39.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 1 librados: starting objecter 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 1 Processor -- start 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 1 -- start start 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 1 librados: setting wanted keys 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 1 librados: calling monclient init 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.201+0000 7f62c1e8d640 10 monclient: init 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f62bc0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f62bc13f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f62bc1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc12f840 con 0x7f62bc0a98f0 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc13bd10 con 0x7f62bc13f710 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62bc0b78c0 con 0x7f62bc1430c0 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:39.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.202+0000 7f62c1e8d640 10 monclient(hunting): authenticate will time out at 7910.490762s 2024-03-25T18:34:39.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.209+0000 7f62baffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f62bc13f710 0x7f62bc1daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:33692/0 (socket says 172.21.15.102:33692) 2024-03-25T18:34:39.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.209+0000 7f62baffd640 1 -- 172.21.15.102:0/3816784885 learned_addr learned my addr 172.21.15.102:0/3816784885 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:39.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.210+0000 7f62a3fff640 10 client.?.objecter ms_handle_connect 0x7f62bc13f710 2024-03-25T18:34:39.212 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.210+0000 7f62a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.216+0000 7f62a3fff640 10 client.?.objecter ms_handle_connect 0x7f62bc1430c0 2024-03-25T18:34:39.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.216+0000 7f62a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.217+0000 7f62a3fff640 10 client.?.objecter ms_handle_connect 0x7f62bc0a98f0 2024-03-25T18:34:39.217 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.217+0000 7f62a3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:39.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.225+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657122665 0 0) 0x7f62bc13bd10 con 0x7f62bc13f710 2024-03-25T18:34:39.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.225+0000 7f62a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.225+0000 7f62a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.225+0000 7f62a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.225+0000 7f62a3fff640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:34:39.226 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.225+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a8004eb0 con 0x7f62bc13f710 2024-03-25T18:34:39.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.229+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319895923 0 0) 0x7f62bc12f840 con 0x7f62bc0a98f0 2024-03-25T18:34:39.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.229+0000 7f62a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.229+0000 7f62a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.229+0000 7f62a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.229+0000 7f62a3fff640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:34:39.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.229+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc13bd10 con 0x7f62bc0a98f0 2024-03-25T18:34:39.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.231+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000164976 0 0) 0x7f62bc0b78c0 con 0x7f62bc1430c0 2024-03-25T18:34:39.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.231+0000 7f62a3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:34:39.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.231+0000 7f62a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:39.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.231+0000 7f62a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.231+0000 7f62a3fff640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:34:39.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.231+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc12f840 con 0x7f62bc1430c0 2024-03-25T18:34:39.248 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.247+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2304434943 0 0) 0x7f62bc12f840 con 0x7f62bc1430c0 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.247+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62bc0b78c0 con 0x7f62bc1430c0 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.247+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f62b4001f10 con 0x7f62bc1430c0 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.247+0000 7f62a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.247+0000 7f62a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.247+0000 7f62a3fff640 10 monclient(hunting): dump: 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:39.249 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.247+0000 7f62a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.247+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 352990398 0 0) 0x7f62bc13bd10 con 0x7f62bc0a98f0 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.248+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62bc12f840 con 0x7f62bc0a98f0 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.248+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f62ac001950 con 0x7f62bc0a98f0 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.248+0000 7f62a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.248+0000 7f62a3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.248+0000 7f62a3fff640 10 monclient(hunting): dump: 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:39.250 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.248+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1406307245 0 0) 0x7f62a8004eb0 con 0x7f62bc13f710 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.248+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62bc13bd10 con 0x7f62bc13f710 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.249+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f62b0001a10 con 0x7f62bc13f710 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.249+0000 7f62a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.249+0000 7f62a3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.249+0000 7f62a3fff640 10 monclient(hunting): dump: 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:39.252 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.261+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4170521390 0 0) 0x7f62bc0b78c0 con 0x7f62bc1430c0 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 >> v1:172.21.15.146:6789/0 conn(0x7f62bc13f710 legacy=0x7f62bc1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 >> v1:172.21.15.102:6789/0 conn(0x7f62bc0a98f0 legacy=0x7f62bc12d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 1 monclient: found mon.c 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc0b7550 con 0x7f62bc1430c0 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 10 monclient: _finish_auth 0 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 5 monclient: authenticate success, global_id 4601 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 10 monclient: _renew_subs 2024-03-25T18:34:39.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc12e1b0 con 0x7f62bc1430c0 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 10 monclient: _renew_subs 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc13c760 con 0x7f62bc1430c0 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.262+0000 7f62c1e8d640 1 librados: init done 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.263+0000 7f62c1e8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.263+0000 7f62c1e8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.263+0000 7f62c1e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:39.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.263+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62bc0b7a50 con 0x7f62bc1430c0 2024-03-25T18:34:39.555 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.554+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62b4001420 con 0x7f62bc1430c0 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.554+0000 7f62a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.554+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f62b4003c20 con 0x7f62bc1430c0 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.554+0000 7f62a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.554+0000 7f62a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.554+0000 7f62a3fff640 10 monclient: dump: 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:39.556 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:39.557 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:39.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.554+0000 7f62b9ffb640 4 set_mon_vals no callback set 2024-03-25T18:34:39.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.555+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f62b4019c30 con 0x7f62bc1430c0 2024-03-25T18:34:39.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.555+0000 7f62bb7fe640 10 monclient: get_auth_request con 0x7f62a806e0a0 auth_method 0 2024-03-25T18:34:39.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.556+0000 7f62a3fff640 10 client.4601.objecter ms_handle_connect 0x7f62a806e0a0 2024-03-25T18:34:39.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.563+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f62b40538b0 con 0x7f62bc1430c0 2024-03-25T18:34:39.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.563+0000 7f62a3fff640 10 client.4601.objecter ms_dispatch 0x7f62bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:39.564 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.563+0000 7f62a3fff640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:39.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.563+0000 7f62a3fff640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:39.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.564+0000 7f62a3fff640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T18:34:39.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.564+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62b404f1f0 con 0x7f62bc1430c0 2024-03-25T18:34:39.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.564+0000 7f62a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:39.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.564+0000 7f62a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:39.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.774+0000 7f62c1e8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.774+0000 7f62c1e8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.774+0000 7f62c1e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:39.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.774+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62bc0b76e0 con 0x7f62bc1430c0 2024-03-25T18:34:39.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.805+0000 7f62a3fff640 1 -- 172.21.15.102:0/3816784885 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f62b40512f0 con 0x7f62bc1430c0 2024-03-25T18:34:39.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.805+0000 7f62a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:39.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.805+0000 7f62a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:39.807 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:34:39.808 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 10 librados: watch_flush enter 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 10 librados: watch_flush exit 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 10 client.4601.objecter successfully canceled tick 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 >> v1:172.21.15.146:6800/3848252714 conn(0x7f62a806e0a0 legacy=0x7f62a8070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 10 monclient: shutdown 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 >> v1:172.21.15.102:6790/0 conn(0x7f62bc1430c0 legacy=0x7f62bc1db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 shutdown_connections 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 >> 172.21.15.102:0/3816784885 conn(0x7f62bc0b7ea0 msgr2=0x7f62bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 shutdown_connections 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 1 -- 172.21.15.102:0/3816784885 wait complete. 2024-03-25T18:34:39.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:39.808+0000 7f62c1e8d640 1 librados: shutdown 2024-03-25T18:34:39.823 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:46.629 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:46.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.741+0000 7f3ba431e640 10 monclient: get_monmap_and_config 2024-03-25T18:34:46.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.741+0000 7f3ba431e640 10 monclient: build_initial_monmap 2024-03-25T18:34:46.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.741+0000 7f3ba431e640 10 monclient: monmap: 2024-03-25T18:34:46.743 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:46.743 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:46.744 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:34:46.743585+0000 2024-03-25T18:34:46.744 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:34:46.743585+0000 2024-03-25T18:34:46.744 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:46.744 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:46.744 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:46.744 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:34:46.744 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:34:46.744 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:46.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.742+0000 7f3ba431e640 1 Processor -- start 2024-03-25T18:34:46.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.742+0000 7f3ba431e640 1 -- start start 2024-03-25T18:34:46.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.742+0000 7f3ba431e640 10 monclient: init 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 10 monclient(hunting): picked mon.noname-c con 0x7f3b9c0b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 10 monclient(hunting): picked mon.noname-b con 0x7f3b9c0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 10 monclient(hunting): picked mon.noname-a con 0x7f3b9c14be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c0af940 con 0x7f3b9c14be90 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c0b0a70 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c0b0d50 con 0x7f3b9c0b1730 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:46.746 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.743+0000 7f3ba431e640 10 monclient(hunting): authenticate will time out at 7918.032376s 2024-03-25T18:34:46.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.753+0000 7f3ba20ba640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3b9c0b4d60 0x7f3b9c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:57154/0 (socket says 172.21.15.102:57154) 2024-03-25T18:34:46.754 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.753+0000 7f3ba20ba640 1 -- 172.21.15.102:0/4094686337 learned_addr learned my addr 172.21.15.102:0/4094686337 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:46.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.766+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890589481 0 0) 0x7f3b9c0b0a70 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.766+0000 7f3ba10b8640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.766+0000 7f3ba10b8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.766+0000 7f3ba10b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.766+0000 7f3ba10b8640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:34:46.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.766+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b88003520 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.771+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180683941 0 0) 0x7f3b9c0b0d50 con 0x7f3b9c0b1730 2024-03-25T18:34:46.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.771+0000 7f3ba10b8640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.771+0000 7f3ba10b8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.771+0000 7f3ba10b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.771+0000 7f3ba10b8640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:34:46.773 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.771+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9c0b0a70 con 0x7f3b9c0b1730 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284907623 0 0) 0x7f3b9c0af940 con 0x7f3b9c14be90 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9c0b0d50 con 0x7f3b9c14be90 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1938767014 0 0) 0x7f3b88003520 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 >> v1:172.21.15.102:6790/0 conn(0x7f3b9c0b1730 legacy=0x7f3b9c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 >> v1:172.21.15.102:6789/0 conn(0x7f3b9c14be90 legacy=0x7f3b9c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient: _finish_hunting 0 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 1 monclient: found mon.noname-c 2024-03-25T18:34:46.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c0b1030 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient: _finish_auth 0 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient: _check_auth_tickets 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3b90001690 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba431e640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba431e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.772+0000 7f3ba10b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.773+0000 7f3ba10b8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.773+0000 7f3ba10b8640 10 monclient: dump: 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:46.776 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:46.777 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.777 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:46.777 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:46.777 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:46.777 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:46.777 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:46.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.773+0000 7f3ba431e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b900017f0 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.791 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba10b8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba10b8640 1 -- 172.21.15.102:0/4094686337 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3b90001c00 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba431e640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba431e640 4 set_mon_vals no callback set 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba10b8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba10b8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba10b8640 10 monclient: dump: 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:46.792 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:46.793 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:46.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba431e640 10 monclient: shutdown 2024-03-25T18:34:46.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba431e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:46.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.790+0000 7f3ba431e640 1 -- 172.21.15.102:0/4094686337 >> v1:172.21.15.146:6789/0 conn(0x7f3b9c0b4d60 legacy=0x7f3b9c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.791+0000 7f3ba431e640 1 -- 172.21.15.102:0/4094686337 shutdown_connections 2024-03-25T18:34:46.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.791+0000 7f3ba431e640 1 -- 172.21.15.102:0/4094686337 >> 172.21.15.102:0/4094686337 conn(0x7f3b9c0617a0 msgr2=0x7f3b9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:46.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.791+0000 7f3ba431e640 1 -- 172.21.15.102:0/4094686337 shutdown_connections 2024-03-25T18:34:46.794 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.791+0000 7f3ba431e640 1 -- 172.21.15.102:0/4094686337 wait complete. 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 10 monclient: build_initial_monmap 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 10 monclient: monmap: 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 1 librados: starting msgr at 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 1 librados: starting objecter 2024-03-25T18:34:46.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 1 Processor -- start 2024-03-25T18:34:46.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 1 -- start start 2024-03-25T18:34:46.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 1 librados: setting wanted keys 2024-03-25T18:34:46.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 1 librados: calling monclient init 2024-03-25T18:34:46.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.792+0000 7f3ba431e640 10 monclient: init 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 10 monclient(hunting): picked mon.noname-c con 0x7f3b9c0b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 10 monclient(hunting): picked mon.noname-a con 0x7f3b9c0b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 10 monclient(hunting): picked mon.noname-b con 0x7f3b9c14be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c1440c0 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c13f020 con 0x7f3b9c14be90 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c136d30 con 0x7f3b9c0b1730 2024-03-25T18:34:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:46.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.793+0000 7f3ba431e640 10 monclient(hunting): authenticate will time out at 7918.082713s 2024-03-25T18:34:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.803+0000 7f3ba20ba640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f3b9c0b1730 0x7f3b9c1439b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:32842/0 (socket says 172.21.15.102:32842) 2024-03-25T18:34:46.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.803+0000 7f3ba20ba640 1 -- 172.21.15.102:0/1336605796 learned_addr learned my addr 172.21.15.102:0/1336605796 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:46.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.803+0000 7f3b8effd640 10 client.?.objecter ms_handle_connect 0x7f3b9c0b1730 2024-03-25T18:34:46.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.803+0000 7f3b8effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.814+0000 7f3b8effd640 10 client.?.objecter ms_handle_connect 0x7f3b9c0b4d60 2024-03-25T18:34:46.816 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.814+0000 7f3b8effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.815+0000 7f3b8effd640 10 client.?.objecter ms_handle_connect 0x7f3b9c14be90 2024-03-25T18:34:46.817 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.815+0000 7f3b8effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:46.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.819+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743443404 0 0) 0x7f3b9c136d30 con 0x7f3b9c0b1730 2024-03-25T18:34:46.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.819+0000 7f3b8effd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.819+0000 7f3b8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.821 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.819+0000 7f3b8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.819+0000 7f3b8effd640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:34:46.822 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.819+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b80004eb0 con 0x7f3b9c0b1730 2024-03-25T18:34:46.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.825+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721913396 0 0) 0x7f3b9c13f020 con 0x7f3b9c14be90 2024-03-25T18:34:46.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.825+0000 7f3b8effd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.825+0000 7f3b8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.825+0000 7f3b8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.825+0000 7f3b8effd640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:34:46.827 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.825+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9c136d30 con 0x7f3b9c14be90 2024-03-25T18:34:46.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.827+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372259361 0 0) 0x7f3b9c1440c0 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.827+0000 7f3b8effd640 10 monclient(hunting): _negotiate 2024-03-25T18:34:46.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.827+0000 7f3b8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:46.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.827+0000 7f3b8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:46.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.827+0000 7f3b8effd640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:34:46.829 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.827+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9c13f020 con 0x7f3b9c0b4d60 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.833+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2118147787 0 0) 0x7f3b80004eb0 con 0x7f3b9c0b1730 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.833+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b9c1440c0 con 0x7f3b9c0b1730 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.833+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3b90001bc0 con 0x7f3b9c0b1730 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.833+0000 7f3b8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.833+0000 7f3b8effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.833+0000 7f3b8effd640 10 monclient(hunting): dump: 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.835 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:46.836 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:46.836 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:46.836 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:46.836 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:46.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.833+0000 7f3b8effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:46.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.840+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2642434989 0 0) 0x7f3b9c136d30 con 0x7f3b9c14be90 2024-03-25T18:34:46.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.840+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b80004eb0 con 0x7f3b9c14be90 2024-03-25T18:34:46.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.840+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3b94001b50 con 0x7f3b9c14be90 2024-03-25T18:34:46.842 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.840+0000 7f3b8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.840+0000 7f3b8effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.840+0000 7f3b8effd640 10 monclient(hunting): dump: 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:46.843 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba20ba640 1 -- 172.21.15.102:0/1336605796 >> v1:172.21.15.102:6790/0 conn(0x7f3b9c0b1730 legacy=0x7f3b9c1439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba20ba640 1 -- 172.21.15.102:0/1336605796 >> v1:172.21.15.102:6790/0 conn(0x7f3b9c0b1730 legacy=0x7f3b9c1439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba20ba640 1 --1- 172.21.15.102:0/1336605796 >> v1:172.21.15.102:6790/0 conn(0x7f3b9c0b1730 0x7f3b9c1439b0 :-1 s=OPENED pgs=65082 cs=1 l=1).handle_message read tag failed 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba20ba640 1 --1- 172.21.15.102:0/1336605796 >> v1:172.21.15.102:6790/0 conn(0x7f3b9c0b1730 0x7f3b9c1439b0 :-1 s=OPENED pgs=65082 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.102:6790/0 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2191999746 0 0) 0x7f3b80004eb0 con 0x7f3b9c14be90 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 >> v1:172.21.15.102:6790/0 conn(0x7f3b9c0b1730 legacy=0x7f3b9c1439b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 >> v1:172.21.15.102:6789/0 conn(0x7f3b9c0b4d60 legacy=0x7f3b9c1d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 10 monclient: _finish_hunting 0 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 1 monclient: found mon.b 2024-03-25T18:34:46.850 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c136ec0 con 0x7f3b9c14be90 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 10 monclient: _finish_auth 0 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 10 monclient: _check_auth_tickets 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3b8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 10 monclient: _renew_subs 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c1e33d0 con 0x7f3b9c14be90 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 10 monclient: _renew_subs 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c1e3740 con 0x7f3b9c14be90 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.847+0000 7f3ba431e640 1 librados: init done 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.848+0000 7f3ba431e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:46.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.848+0000 7f3ba431e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:46.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.848+0000 7f3ba431e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:46.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:46.848+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c1d79a0 con 0x7f3b9c14be90 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.131+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b94001fe0 con 0x7f3b9c14be90 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.131+0000 7f3b8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.131+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3b94003b10 con 0x7f3b9c14be90 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.131+0000 7f3b8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.131+0000 7f3b8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.131+0000 7f3b8effd640 10 monclient: dump: 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:47.133 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:47.134 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:47.134 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:47.134 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:47.134 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:47.134 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:47.134 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:47.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.131+0000 7f3ba08b7640 4 set_mon_vals no callback set 2024-03-25T18:34:47.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.132+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f3b94003d30 con 0x7f3b9c14be90 2024-03-25T18:34:47.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.132+0000 7f3ba20ba640 10 monclient: get_auth_request con 0x7f3b8006df70 auth_method 0 2024-03-25T18:34:47.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.133+0000 7f3b8effd640 10 client.4627.objecter ms_handle_connect 0x7f3b8006df70 2024-03-25T18:34:47.138 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.136+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f3b940537d0 con 0x7f3b9c14be90 2024-03-25T18:34:47.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.136+0000 7f3b8effd640 10 client.4627.objecter ms_dispatch 0x7f3b9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:47.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.136+0000 7f3b8effd640 3 client.4627.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:47.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.136+0000 7f3b8effd640 3 client.4627.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:47.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.136+0000 7f3b8effd640 20 client.4627.objecter dump_active .. 0 homeless 2024-03-25T18:34:47.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.137+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b9404f110 con 0x7f3b9c14be90 2024-03-25T18:34:47.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.137+0000 7f3b8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:47.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.137+0000 7f3b8effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:47.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.373+0000 7f3ba431e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.373+0000 7f3ba431e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.376 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.373+0000 7f3ba431e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:34:47.376 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.373+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b9c1401f0 con 0x7f3b9c14be90 2024-03-25T18:34:47.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.402+0000 7f3b8effd640 1 -- 172.21.15.102:0/1336605796 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f3b94051210 con 0x7f3b9c14be90 2024-03-25T18:34:47.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.402+0000 7f3b8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:47.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.402+0000 7f3b8effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:47.404 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:34:47.404 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 10 librados: watch_flush enter 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 10 librados: watch_flush exit 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 20 client.4627.objecter shutdown clearing up homeless session... 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 10 client.4627.objecter successfully canceled tick 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 >> v1:172.21.15.146:6800/3848252714 conn(0x7f3b8006df70 legacy=0x7f3b80070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 10 monclient: shutdown 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 >> v1:172.21.15.146:6789/0 conn(0x7f3b9c14be90 legacy=0x7f3b9c1d8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 shutdown_connections 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 >> 172.21.15.102:0/1336605796 conn(0x7f3b9c0617a0 msgr2=0x7f3b9c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 shutdown_connections 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.404+0000 7f3ba431e640 1 -- 172.21.15.102:0/1336605796 wait complete. 2024-03-25T18:34:47.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:47.405+0000 7f3ba431e640 1 librados: shutdown 2024-03-25T18:34:47.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:34:49.676 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:05.911 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.728 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:05.963 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.758 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:05.994 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.779 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:06.015 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.782 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:05.995 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.791 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:06.004 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.795 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:06.031 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.823 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:06.059 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:49.837 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:06.050 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:34:54.224 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.332+0000 7f25ad44a640 10 monclient: get_monmap_and_config 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.332+0000 7f25ad44a640 10 monclient: build_initial_monmap 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.332+0000 7f25ad44a640 10 monclient: monmap: 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:34:54.334059+0000 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:34:54.334059+0000 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:34:54.334 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:34:54.335 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.332+0000 7f25ad44a640 1 Processor -- start 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.333+0000 7f25ad44a640 1 -- start start 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.333+0000 7f25ad44a640 10 monclient: init 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.333+0000 7f25ad44a640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.333+0000 7f25ad44a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.333+0000 7f25ad44a640 10 monclient(hunting): picked mon.noname-a con 0x7f25a80a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.333+0000 7f25ad44a640 10 monclient(hunting): picked mon.noname-c con 0x7f25a813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.333+0000 7f25ad44a640 10 monclient(hunting): picked mon.noname-b con 0x7f25a81430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.334+0000 7f25ad44a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a8142520 con 0x7f25a80a98f0 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.334+0000 7f25ad44a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80b28a0 con 0x7f25a813f710 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.334+0000 7f25ad44a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80b2a30 con 0x7f25a81430c0 2024-03-25T18:34:54.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.334+0000 7f25ad44a640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:54.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.334+0000 7f25ad44a640 10 monclient(hunting): authenticate will time out at 7925.623185s 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.340+0000 7f25a67fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f25a813f710 0x7f25a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:37502/0 (socket says 172.21.15.102:37502) 2024-03-25T18:34:54.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.340+0000 7f25a67fc640 1 -- 172.21.15.102:0/2405289534 learned_addr learned my addr 172.21.15.102:0/2405289534 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:54.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.352+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643348618 0 0) 0x7f25a80b28a0 con 0x7f25a813f710 2024-03-25T18:34:54.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.352+0000 7f25a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.354 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.352+0000 7f25a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.354 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.352+0000 7f25a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.354 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.352+0000 7f25a5ffb640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:34:54.354 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.352+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2590003520 con 0x7f25a813f710 2024-03-25T18:34:54.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.362+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336465176 0 0) 0x7f25a8142520 con 0x7f25a80a98f0 2024-03-25T18:34:54.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.362+0000 7f25a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.362+0000 7f25a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.362+0000 7f25a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.362+0000 7f25a5ffb640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:34:54.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.362+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a80b28a0 con 0x7f25a80a98f0 2024-03-25T18:34:54.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3656344992 0 0) 0x7f2590003520 con 0x7f25a813f710 2024-03-25T18:34:54.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 >> v1:172.21.15.102:6790/0 conn(0x7f25a81430c0 legacy=0x7f25a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 >> v1:172.21.15.102:6789/0 conn(0x7f25a80a98f0 legacy=0x7f25a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:34:54.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a80b44c0 con 0x7f25a813f710 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f259c001690 con 0x7f25a813f710 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25ad44a640 5 monclient: authenticate success, global_id 4630 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25ad44a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25a5ffb640 10 monclient: dump: 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:54.367 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:54.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.364+0000 7f25ad44a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:34:54.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.379+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f259c0017f0 con 0x7f25a813f710 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.379+0000 7f25a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.379+0000 7f25a5ffb640 1 -- 172.21.15.102:0/2405289534 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f259c001c00 con 0x7f25a813f710 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.379+0000 7f25ad44a640 10 monclient: get_monmap_and_config success 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.379+0000 7f25ad44a640 4 set_mon_vals no callback set 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.379+0000 7f25a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.379+0000 7f25a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.379+0000 7f25a5ffb640 10 monclient: dump: 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:54.381 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:54.382 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:54.382 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:54.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.380+0000 7f25ad44a640 10 monclient: shutdown 2024-03-25T18:34:54.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.380+0000 7f25ad44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:54.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.380+0000 7f25ad44a640 1 -- 172.21.15.102:0/2405289534 >> v1:172.21.15.146:6789/0 conn(0x7f25a813f710 legacy=0x7f25a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.380+0000 7f25ad44a640 1 -- 172.21.15.102:0/2405289534 shutdown_connections 2024-03-25T18:34:54.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.380+0000 7f25ad44a640 1 -- 172.21.15.102:0/2405289534 >> 172.21.15.102:0/2405289534 conn(0x7f25a80b7ea0 msgr2=0x7f25a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:54.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.380+0000 7f25ad44a640 1 -- 172.21.15.102:0/2405289534 shutdown_connections 2024-03-25T18:34:54.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.380+0000 7f25ad44a640 1 -- 172.21.15.102:0/2405289534 wait complete. 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 10 monclient: build_initial_monmap 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 10 monclient: monmap: 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 1 librados: starting msgr at 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 1 librados: starting objecter 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 1 Processor -- start 2024-03-25T18:34:54.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 1 -- start start 2024-03-25T18:34:54.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 1 librados: setting wanted keys 2024-03-25T18:34:54.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 1 librados: calling monclient init 2024-03-25T18:34:54.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.381+0000 7f25ad44a640 10 monclient: init 2024-03-25T18:34:54.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 10 monclient: _reopen_session rank -1 2024-03-25T18:34:54.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:34:54.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 10 monclient(hunting): picked mon.noname-b con 0x7f25a80a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:34:54.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 10 monclient(hunting): picked mon.noname-a con 0x7f25a813f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:34:54.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 10 monclient(hunting): picked mon.noname-c con 0x7f25a81430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:34:54.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a8138a50 con 0x7f25a813f710 2024-03-25T18:34:54.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a80ab9e0 con 0x7f25a80a98f0 2024-03-25T18:34:54.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a81d7880 con 0x7f25a81430c0 2024-03-25T18:34:54.388 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 10 monclient(hunting): _renew_subs 2024-03-25T18:34:54.388 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.382+0000 7f25ad44a640 10 monclient(hunting): authenticate will time out at 7925.671537s 2024-03-25T18:34:54.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.400+0000 7f25a77fe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f25a81430c0 0x7f25a81e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:37398/0 (socket says 172.21.15.102:37398) 2024-03-25T18:34:54.402 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.401+0000 7f25a77fe640 1 -- 172.21.15.102:0/2241103979 learned_addr learned my addr 172.21.15.102:0/2241103979 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:34:54.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.405+0000 7f258f7fe640 10 client.?.objecter ms_handle_connect 0x7f25a81430c0 2024-03-25T18:34:54.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.405+0000 7f258f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.406+0000 7f258f7fe640 10 client.?.objecter ms_handle_connect 0x7f25a813f710 2024-03-25T18:34:54.407 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.406+0000 7f258f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.406+0000 7f258f7fe640 10 client.?.objecter ms_handle_connect 0x7f25a80a98f0 2024-03-25T18:34:54.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.406+0000 7f258f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:34:54.413 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.412+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537703779 0 0) 0x7f25a81d7880 con 0x7f25a81430c0 2024-03-25T18:34:54.413 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.412+0000 7f258f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.413 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.412+0000 7f258f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.413 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.412+0000 7f258f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.412+0000 7f258f7fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:34:54.414 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.412+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2594004eb0 con 0x7f25a81430c0 2024-03-25T18:34:54.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.417+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352505168 0 0) 0x7f25a80ab9e0 con 0x7f25a80a98f0 2024-03-25T18:34:54.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.417+0000 7f258f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.417+0000 7f258f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.417+0000 7f258f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.417+0000 7f258f7fe640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:34:54.419 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.417+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a81d7880 con 0x7f25a80a98f0 2024-03-25T18:34:54.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.421+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644619375 0 0) 0x7f25a8138a50 con 0x7f25a813f710 2024-03-25T18:34:54.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.421+0000 7f258f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:34:54.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.421+0000 7f258f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:34:54.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.421+0000 7f258f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:34:54.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.421+0000 7f258f7fe640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:34:54.423 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.421+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a80ab9e0 con 0x7f25a813f710 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.429+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4041635678 0 0) 0x7f2594004eb0 con 0x7f25a81430c0 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.429+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25a8138a50 con 0x7f25a81430c0 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.429+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f25a0001c90 con 0x7f25a81430c0 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.429+0000 7f258f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.429+0000 7f258f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.429+0000 7f258f7fe640 10 monclient(hunting): dump: 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:54.431 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:54.432 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:54.432 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:54.432 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:54.432 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.429+0000 7f258f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.437+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4186000369 0 0) 0x7f25a81d7880 con 0x7f25a80a98f0 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.437+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2594004eb0 con 0x7f25a80a98f0 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.437+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2598001930 con 0x7f25a80a98f0 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.437+0000 7f258f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.437+0000 7f258f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.437+0000 7f258f7fe640 10 monclient(hunting): dump: 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.439 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.440 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.440 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:54.440 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:54.440 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:54.440 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:54.440 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:54.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1290628050 0 0) 0x7f25a8138a50 con 0x7f25a81430c0 2024-03-25T18:34:54.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 >> v1:172.21.15.146:6789/0 conn(0x7f25a80a98f0 legacy=0x7f25a8138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 >> v1:172.21.15.102:6789/0 conn(0x7f25a813f710 legacy=0x7f25a81d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:54.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:34:54.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 1 monclient: found mon.c 2024-03-25T18:34:54.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:34:54.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:54.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a81d84b0 con 0x7f25a81430c0 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f258f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 5 monclient: authenticate success, global_id 4613 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 10 monclient: _renew_subs 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a81d51f0 con 0x7f25a81430c0 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 10 monclient: _renew_subs 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a8138e60 con 0x7f25a81430c0 2024-03-25T18:34:54.446 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.443+0000 7f25ad44a640 1 librados: init done 2024-03-25T18:34:54.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.444+0000 7f25ad44a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:54.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.444+0000 7f25ad44a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:54.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.444+0000 7f25ad44a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:54.447 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.444+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a81d76e0 con 0x7f25a81430c0 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.749+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25a0003990 con 0x7f25a81430c0 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.749+0000 7f258f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.749+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f25a0003b40 con 0x7f25a81430c0 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.749+0000 7f258f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.749+0000 7f258f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.749+0000 7f258f7fe640 10 monclient: dump: 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.751 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.749+0000 7f25a57fa640 4 set_mon_vals no callback set 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.750+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f25a0003d60 con 0x7f25a81430c0 2024-03-25T18:34:54.752 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.750+0000 7f25a6ffd640 10 monclient: get_auth_request con 0x7f259406df70 auth_method 0 2024-03-25T18:34:54.753 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.751+0000 7f258f7fe640 10 client.4613.objecter ms_handle_connect 0x7f259406df70 2024-03-25T18:34:54.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.756+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f25a0054950 con 0x7f25a81430c0 2024-03-25T18:34:54.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.757+0000 7f258f7fe640 10 client.4613.objecter ms_dispatch 0x7f25a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:34:54.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.757+0000 7f258f7fe640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:34:54.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.757+0000 7f258f7fe640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:34:54.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.757+0000 7f258f7fe640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-25T18:34:54.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.757+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25a0050460 con 0x7f25a81430c0 2024-03-25T18:34:54.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.757+0000 7f258f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:34:54.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:54.757+0000 7f258f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:34:55.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.011+0000 7f25ad44a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:55.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.011+0000 7f25ad44a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:55.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.011+0000 7f25ad44a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:34:55.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.011+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25a80627b0 con 0x7f25a81430c0 2024-03-25T18:34:55.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.047+0000 7f258f7fe640 1 -- 172.21.15.102:0/2241103979 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f25a0052390 con 0x7f25a81430c0 2024-03-25T18:34:55.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.047+0000 7f258f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:34:55.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.047+0000 7f258f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:34:55.049 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:34:55.050 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 10 librados: watch_flush enter 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 10 librados: watch_flush exit 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 10 client.4613.objecter successfully canceled tick 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 >> v1:172.21.15.146:6800/3848252714 conn(0x7f259406df70 legacy=0x7f2594070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 10 monclient: shutdown 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 >> v1:172.21.15.102:6790/0 conn(0x7f25a81430c0 legacy=0x7f25a81e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 shutdown_connections 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 >> 172.21.15.102:0/2241103979 conn(0x7f25a80b7ea0 msgr2=0x7f25a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 shutdown_connections 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 1 -- 172.21.15.102:0/2241103979 wait complete. 2024-03-25T18:34:55.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:34:55.050+0000 7f25ad44a640 1 librados: shutdown 2024-03-25T18:34:55.064 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:01.870 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.989+0000 7ffadbe82640 10 monclient: get_monmap_and_config 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.989+0000 7ffadbe82640 10 monclient: build_initial_monmap 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.989+0000 7ffadbe82640 10 monclient: monmap: 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:35:01.989998+0000 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:35:01.989998+0000 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:01.990 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:35:01.991 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:35:01.991 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.990+0000 7ffadbe82640 1 Processor -- start 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.990+0000 7ffadbe82640 1 -- start start 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.990+0000 7ffadbe82640 10 monclient: init 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 10 monclient(hunting): picked mon.noname-b con 0x7ffad40a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 10 monclient(hunting): picked mon.noname-a con 0x7ffad413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 10 monclient(hunting): picked mon.noname-c con 0x7ffad41430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad4142520 con 0x7ffad413f710 2024-03-25T18:35:01.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad40b28a0 con 0x7ffad41430c0 2024-03-25T18:35:01.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad40b2a30 con 0x7ffad40a98f0 2024-03-25T18:35:01.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:01.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.991+0000 7ffadbe82640 10 monclient(hunting): authenticate will time out at 7933.279267s 2024-03-25T18:35:01.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.998+0000 7ffada41f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ffad41430c0 0x7ffad414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:33908/0 (socket says 172.21.15.102:33908) 2024-03-25T18:35:01.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:01.998+0000 7ffada41f640 1 -- 172.21.15.102:0/1374832337 learned_addr learned my addr 172.21.15.102:0/1374832337 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:02.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.011+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236005634 0 0) 0x7ffad40b28a0 con 0x7ffad41430c0 2024-03-25T18:35:02.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.011+0000 7ffad8c1c640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.011+0000 7ffad8c1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.011+0000 7ffad8c1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.011+0000 7ffad8c1c640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:35:02.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.011+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac0003520 con 0x7ffad41430c0 2024-03-25T18:35:02.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.025+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512556356 0 0) 0x7ffad40b2a30 con 0x7ffad40a98f0 2024-03-25T18:35:02.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.025+0000 7ffad8c1c640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.025+0000 7ffad8c1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.025+0000 7ffad8c1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.025+0000 7ffad8c1c640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:35:02.027 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.025+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad40b28a0 con 0x7ffad40a98f0 2024-03-25T18:35:02.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2948292825 0 0) 0x7ffac0003520 con 0x7ffad41430c0 2024-03-25T18:35:02.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 >> v1:172.21.15.102:6790/0 conn(0x7ffad40a98f0 legacy=0x7ffad40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 >> v1:172.21.15.102:6789/0 conn(0x7ffad413f710 legacy=0x7ffad41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 10 monclient: _finish_hunting 0 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 1 monclient: found mon.noname-c 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad40b44c0 con 0x7ffad41430c0 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 10 monclient: _finish_auth 0 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 10 monclient: _check_auth_tickets 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ffacc001a10 con 0x7ffad41430c0 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffad8c1c640 10 monclient: dump: 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:02.029 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffadbe82640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:35:02.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.027+0000 7ffadbe82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffacc001b70 con 0x7ffad41430c0 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffad8c1c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffad8c1c640 1 -- 172.21.15.102:0/1374832337 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ffacc001f80 con 0x7ffad41430c0 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffad8c1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffad8c1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffad8c1c640 10 monclient: dump: 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:02.048 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:02.049 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:02.049 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:02.049 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:02.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffadbe82640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:02.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffadbe82640 4 set_mon_vals no callback set 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffadbe82640 10 monclient: shutdown 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffadbe82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.047+0000 7ffadbe82640 1 -- 172.21.15.102:0/1374832337 >> v1:172.21.15.146:6789/0 conn(0x7ffad41430c0 legacy=0x7ffad414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.048+0000 7ffadbe82640 1 -- 172.21.15.102:0/1374832337 shutdown_connections 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.048+0000 7ffadbe82640 1 -- 172.21.15.102:0/1374832337 >> 172.21.15.102:0/1374832337 conn(0x7ffad40b7ea0 msgr2=0x7ffad40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.048+0000 7ffadbe82640 1 -- 172.21.15.102:0/1374832337 shutdown_connections 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.048+0000 7ffadbe82640 1 -- 172.21.15.102:0/1374832337 wait complete. 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.048+0000 7ffadbe82640 10 monclient: build_initial_monmap 2024-03-25T18:35:02.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.048+0000 7ffadbe82640 10 monclient: monmap: 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.049+0000 7ffadbe82640 1 librados: starting msgr at 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.049+0000 7ffadbe82640 1 librados: starting objecter 2024-03-25T18:35:02.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.049+0000 7ffadbe82640 1 Processor -- start 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.049+0000 7ffadbe82640 1 -- start start 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.049+0000 7ffadbe82640 1 librados: setting wanted keys 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.049+0000 7ffadbe82640 1 librados: calling monclient init 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.049+0000 7ffadbe82640 10 monclient: init 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.049+0000 7ffadbe82640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 10 monclient(hunting): picked mon.noname-c con 0x7ffad40a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 10 monclient(hunting): picked mon.noname-a con 0x7ffad413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 10 monclient(hunting): picked mon.noname-b con 0x7ffad41430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad40ab920 con 0x7ffad413f710 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad4135140 con 0x7ffad41430c0 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad4134e10 con 0x7ffad40a98f0 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:02.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.050+0000 7ffadbe82640 10 monclient(hunting): authenticate will time out at 7933.338139s 2024-03-25T18:35:02.059 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.058+0000 7ffad941d640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffad413f710 0x7ffad4133ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38118/0 (socket says 172.21.15.102:38118) 2024-03-25T18:35:02.059 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.058+0000 7ffad941d640 1 -- 172.21.15.102:0/970286502 learned_addr learned my addr 172.21.15.102:0/970286502 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:02.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.061+0000 7ffaca7fc640 10 client.?.objecter ms_handle_connect 0x7ffad413f710 2024-03-25T18:35:02.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.061+0000 7ffaca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:02.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.063+0000 7ffaca7fc640 10 client.?.objecter ms_handle_connect 0x7ffad40a98f0 2024-03-25T18:35:02.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.063+0000 7ffaca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:02.066 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.066+0000 7ffaca7fc640 10 client.?.objecter ms_handle_connect 0x7ffad41430c0 2024-03-25T18:35:02.067 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.066+0000 7ffaca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:02.073 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.072+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950222396 0 0) 0x7ffad4134e10 con 0x7ffad40a98f0 2024-03-25T18:35:02.073 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.073+0000 7ffaca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:02.074 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.073+0000 7ffaca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:02.074 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.073+0000 7ffaca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:02.074 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.073+0000 7ffaca7fc640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:35:02.074 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.073+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab8004eb0 con 0x7ffad40a98f0 2024-03-25T18:35:02.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.083+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3358825139 0 0) 0x7ffab8004eb0 con 0x7ffad40a98f0 2024-03-25T18:35:02.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.083+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad4134e10 con 0x7ffad40a98f0 2024-03-25T18:35:02.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.083+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ffad0001970 con 0x7ffad40a98f0 2024-03-25T18:35:02.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.083+0000 7ffaca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.083+0000 7ffaca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:02.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.083+0000 7ffaca7fc640 10 monclient(hunting): dump: 2024-03-25T18:35:02.084 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:02.085 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.083+0000 7ffaca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:02.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1812367393 0 0) 0x7ffad4134e10 con 0x7ffad40a98f0 2024-03-25T18:35:02.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 >> v1:172.21.15.146:6789/0 conn(0x7ffad41430c0 legacy=0x7ffad4135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 >> v1:172.21.15.102:6789/0 conn(0x7ffad413f710 legacy=0x7ffad4133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.144 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 1 monclient: found mon.c 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad4137840 con 0x7ffad40a98f0 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffaca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 5 monclient: authenticate success, global_id 4619 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 10 monclient: _renew_subs 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad4137a90 con 0x7ffad40a98f0 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:02.145 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 10 monclient: _renew_subs 2024-03-25T18:35:02.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:02.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad41347e0 con 0x7ffad40a98f0 2024-03-25T18:35:02.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.143+0000 7ffadbe82640 1 librados: init done 2024-03-25T18:35:02.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.144+0000 7ffadbe82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:02.146 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.144+0000 7ffadbe82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:02.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.144+0000 7ffadbe82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:02.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.144+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad4134fa0 con 0x7ffad40a98f0 2024-03-25T18:35:02.167 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.166+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffad0003610 con 0x7ffad40a98f0 2024-03-25T18:35:02.167 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.166+0000 7ffaca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:02.167 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.166+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7ffad00038c0 con 0x7ffad40a98f0 2024-03-25T18:35:02.167 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.166+0000 7ffaca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.166+0000 7ffaca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.166+0000 7ffaca7fc640 10 monclient: dump: 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:02.168 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:02.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.166+0000 7ffacbfff640 4 set_mon_vals no callback set 2024-03-25T18:35:02.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.167+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7ffad0003ae0 con 0x7ffad40a98f0 2024-03-25T18:35:02.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.167+0000 7ffad941d640 10 monclient: get_auth_request con 0x7ffab806dd00 auth_method 0 2024-03-25T18:35:02.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.167+0000 7ffaca7fc640 10 client.4619.objecter ms_handle_connect 0x7ffab806dd00 2024-03-25T18:35:02.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.169+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7ffad00534f0 con 0x7ffad40a98f0 2024-03-25T18:35:02.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.169+0000 7ffaca7fc640 10 client.4619.objecter ms_dispatch 0x7ffad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:02.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.169+0000 7ffaca7fc640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:02.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.169+0000 7ffaca7fc640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:02.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.169+0000 7ffaca7fc640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T18:35:02.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.467+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffad004f000 con 0x7ffad40a98f0 2024-03-25T18:35:02.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.467+0000 7ffaca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:02.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.467+0000 7ffaca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:02.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.723+0000 7ffadbe82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.723+0000 7ffadbe82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.723+0000 7ffadbe82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:02.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.723+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad414f320 con 0x7ffad40a98f0 2024-03-25T18:35:02.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.754+0000 7ffaca7fc640 1 -- 172.21.15.102:0/970286502 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7ffad0050f30 con 0x7ffad40a98f0 2024-03-25T18:35:02.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.755+0000 7ffaca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:02.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.755+0000 7ffaca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:02.756 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:35:02.756 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:02.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.756+0000 7ffadbe82640 10 librados: watch_flush enter 2024-03-25T18:35:02.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.756+0000 7ffadbe82640 10 librados: watch_flush exit 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.756+0000 7ffadbe82640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.756+0000 7ffadbe82640 10 client.4619.objecter successfully canceled tick 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 >> v1:172.21.15.146:6800/3848252714 conn(0x7ffab806dd00 legacy=0x7ffab80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 10 monclient: shutdown 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 >> v1:172.21.15.102:6790/0 conn(0x7ffad40a98f0 legacy=0x7ffad41385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 shutdown_connections 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 >> 172.21.15.102:0/970286502 conn(0x7ffad40b7ea0 msgr2=0x7ffad413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 shutdown_connections 2024-03-25T18:35:02.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 1 -- 172.21.15.102:0/970286502 wait complete. 2024-03-25T18:35:02.760 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:02.757+0000 7ffadbe82640 1 librados: shutdown 2024-03-25T18:35:02.769 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:09.578 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:09.696 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.695+0000 7fe9063d6640 10 monclient: get_monmap_and_config 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.695+0000 7fe9063d6640 10 monclient: build_initial_monmap 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.695+0000 7fe9063d6640 10 monclient: monmap: 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:35:09.696426+0000 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:35:09.696426+0000 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:09.697 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:35:09.698 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:35:09.698 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:09.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.696+0000 7fe9063d6640 1 Processor -- start 2024-03-25T18:35:09.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.696+0000 7fe9063d6640 1 -- start start 2024-03-25T18:35:09.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.696+0000 7fe9063d6640 10 monclient: init 2024-03-25T18:35:09.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:09.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:09.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9000a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:09.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe90013f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:09.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9001430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:09.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe900142520 con 0x7fe9000a98f0 2024-03-25T18:35:09.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9000b28a0 con 0x7fe9001430c0 2024-03-25T18:35:09.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9000b2a30 con 0x7fe90013f710 2024-03-25T18:35:09.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:09.700 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.697+0000 7fe9063d6640 10 monclient(hunting): authenticate will time out at 7940.985666s 2024-03-25T18:35:09.712 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.711+0000 7fe904973640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe9001430c0 0x7fe90014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:38908/0 (socket says 172.21.15.102:38908) 2024-03-25T18:35:09.712 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.711+0000 7fe904973640 1 -- 172.21.15.102:0/3056584974 learned_addr learned my addr 172.21.15.102:0/3056584974 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:09.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.716+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539617295 0 0) 0x7fe9000b28a0 con 0x7fe9001430c0 2024-03-25T18:35:09.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.716+0000 7fe8feffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.716+0000 7fe8feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.716+0000 7fe8feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.716+0000 7fe8feffd640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:35:09.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.716+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e8003520 con 0x7fe9001430c0 2024-03-25T18:35:09.727 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1621668325 0 0) 0x7fe8e8003520 con 0x7fe9001430c0 2024-03-25T18:35:09.727 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 >> v1:172.21.15.102:6790/0 conn(0x7fe90013f710 legacy=0x7fe9001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:35:09.727 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 >> v1:172.21.15.102:6789/0 conn(0x7fe9000a98f0 legacy=0x7fe9000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.727 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:09.727 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 1 monclient: found mon.noname-c 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000b44c0 con 0x7fe9001430c0 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe8f8001a10 con 0x7fe9001430c0 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe9063d6640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe9063d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.728 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe8feffd640 10 monclient: dump: 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:09.729 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:09.730 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:09.730 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.726+0000 7fe9063d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:09.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.731+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8f8001be0 con 0x7fe9001430c0 2024-03-25T18:35:09.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.731+0000 7fe8feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:09.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe8feffd640 1 -- 172.21.15.102:0/3056584974 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe8f8002030 con 0x7fe9001430c0 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe8feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 4 set_mon_vals no callback set 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe8feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe8feffd640 10 monclient: dump: 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:09.734 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 10 monclient: shutdown 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 1 -- 172.21.15.102:0/3056584974 >> v1:172.21.15.146:6789/0 conn(0x7fe9001430c0 legacy=0x7fe90014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 1 -- 172.21.15.102:0/3056584974 shutdown_connections 2024-03-25T18:35:09.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 1 -- 172.21.15.102:0/3056584974 >> 172.21.15.102:0/3056584974 conn(0x7fe9000b7ea0 msgr2=0x7fe9000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:09.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 1 -- 172.21.15.102:0/3056584974 shutdown_connections 2024-03-25T18:35:09.736 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.732+0000 7fe9063d6640 1 -- 172.21.15.102:0/3056584974 wait complete. 2024-03-25T18:35:09.737 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.733+0000 7fe9063d6640 10 monclient: build_initial_monmap 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.733+0000 7fe9063d6640 10 monclient: monmap: 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:09.738 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.733+0000 7fe9063d6640 1 librados: starting msgr at 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.733+0000 7fe9063d6640 1 librados: starting objecter 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.733+0000 7fe9063d6640 1 Processor -- start 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 1 -- start start 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 1 librados: setting wanted keys 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 1 librados: calling monclient init 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 10 monclient: init 2024-03-25T18:35:09.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9000a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 10 monclient(hunting): picked mon.noname-b con 0x7fe90013f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9001430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe900139d20 con 0x7fe9000a98f0 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90013c270 con 0x7fe90013f710 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90013b380 con 0x7fe9001430c0 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.734+0000 7fe9063d6640 10 monclient(hunting): authenticate will time out at 7941.022750s 2024-03-25T18:35:09.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.737+0000 7fe904973640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fe9001430c0 0x7fe9001e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:43218/0 (socket says 172.21.15.102:43218) 2024-03-25T18:35:09.741 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.737+0000 7fe904973640 1 -- 172.21.15.102:0/3549978322 learned_addr learned my addr 172.21.15.102:0/3549978322 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:09.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.741+0000 7fe8fcff9640 10 client.?.objecter ms_handle_connect 0x7fe9000a98f0 2024-03-25T18:35:09.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.741+0000 7fe8fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:09.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.742+0000 7fe8fcff9640 10 client.?.objecter ms_handle_connect 0x7fe9001430c0 2024-03-25T18:35:09.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.742+0000 7fe8fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:09.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.747+0000 7fe8fcff9640 10 client.?.objecter ms_handle_connect 0x7fe90013f710 2024-03-25T18:35:09.748 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.747+0000 7fe8fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:09.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.748+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470143861 0 0) 0x7fe900139d20 con 0x7fe9000a98f0 2024-03-25T18:35:09.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.748+0000 7fe8fcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.748+0000 7fe8fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.748+0000 7fe8fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.748+0000 7fe8fcff9640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:35:09.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.748+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ec004eb0 con 0x7fe9000a98f0 2024-03-25T18:35:09.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.750+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101867461 0 0) 0x7fe90013b380 con 0x7fe9001430c0 2024-03-25T18:35:09.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.750+0000 7fe8fcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.750+0000 7fe8fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.750+0000 7fe8fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.750+0000 7fe8fcff9640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:35:09.752 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.750+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe900139d20 con 0x7fe9001430c0 2024-03-25T18:35:09.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.766+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869517199 0 0) 0x7fe90013c270 con 0x7fe90013f710 2024-03-25T18:35:09.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.766+0000 7fe8fcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:35:09.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.766+0000 7fe8fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:09.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.766+0000 7fe8fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:09.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.766+0000 7fe8fcff9640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:35:09.767 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.766+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe90013b380 con 0x7fe90013f710 2024-03-25T18:35:09.769 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.768+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1782189847 0 0) 0x7fe900139d20 con 0x7fe9001430c0 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.768+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe90013c270 con 0x7fe9001430c0 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.768+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe8f8001ef0 con 0x7fe9001430c0 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.768+0000 7fe8fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.768+0000 7fe8fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.768+0000 7fe8fcff9640 10 monclient(hunting): dump: 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.770 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:09.771 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:09.771 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:09.771 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:09.771 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:09.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.768+0000 7fe8fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:09.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.770+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668463646 0 0) 0x7fe8ec004eb0 con 0x7fe9000a98f0 2024-03-25T18:35:09.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.770+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe900139d20 con 0x7fe9000a98f0 2024-03-25T18:35:09.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.770+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe8f0001930 con 0x7fe9000a98f0 2024-03-25T18:35:09.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.770+0000 7fe8fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:09.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.770+0000 7fe8fcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:09.772 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.770+0000 7fe8fcff9640 10 monclient(hunting): dump: 2024-03-25T18:35:09.772 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:09.772 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:09.773 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:09.773 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:09.773 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:09.773 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:09.773 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:09.773 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:09.773 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:09.773 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:09.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4124302183 0 0) 0x7fe90013c270 con 0x7fe9001430c0 2024-03-25T18:35:09.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 >> v1:172.21.15.146:6789/0 conn(0x7fe90013f710 legacy=0x7fe90013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 >> v1:172.21.15.102:6789/0 conn(0x7fe9000a98f0 legacy=0x7fe90012f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:09.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:35:09.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 1 monclient: found mon.c 2024-03-25T18:35:09.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:09.780 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9001399b0 con 0x7fe9001430c0 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 10 monclient: _finish_auth 0 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe8fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 5 monclient: authenticate success, global_id 4622 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 10 monclient: _renew_subs 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9000b70b0 con 0x7fe9001430c0 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 10 monclient: _renew_subs 2024-03-25T18:35:09.781 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:09.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9000b7b70 con 0x7fe9001430c0 2024-03-25T18:35:09.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.778+0000 7fe9063d6640 1 librados: init done 2024-03-25T18:35:09.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.779+0000 7fe9063d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:09.782 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.779+0000 7fe9063d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:09.783 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.779+0000 7fe9063d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:09.783 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:09.779+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9000627b0 con 0x7fe9001430c0 2024-03-25T18:35:10.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.107+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8f80020b0 con 0x7fe9001430c0 2024-03-25T18:35:10.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.107+0000 7fe8fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:10.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.107+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fe8f8004fb0 con 0x7fe9001430c0 2024-03-25T18:35:10.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.107+0000 7fe8fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:10.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.107+0000 7fe8fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:10.108 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.107+0000 7fe8fcff9640 10 monclient: dump: 2024-03-25T18:35:10.108 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:10.109 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.107+0000 7fe8fe7fc640 4 set_mon_vals no callback set 2024-03-25T18:35:10.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.108+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fe8f801af80 con 0x7fe9001430c0 2024-03-25T18:35:10.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.108+0000 7fe8fffff640 10 monclient: get_auth_request con 0x7fe8ec06df10 auth_method 0 2024-03-25T18:35:10.110 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.109+0000 7fe8fcff9640 10 client.4622.objecter ms_handle_connect 0x7fe8ec06df10 2024-03-25T18:35:10.115 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.114+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fe8f8054b20 con 0x7fe9001430c0 2024-03-25T18:35:10.115 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.114+0000 7fe8fcff9640 10 client.4622.objecter ms_dispatch 0x7fe900000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:10.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.114+0000 7fe8fcff9640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:10.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.114+0000 7fe8fcff9640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:10.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.114+0000 7fe8fcff9640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-25T18:35:10.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.114+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8f8050630 con 0x7fe9001430c0 2024-03-25T18:35:10.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.114+0000 7fe8fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:10.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.114+0000 7fe8fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:10.165 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:26.401 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.214 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:26.450 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.239 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:26.475 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.282 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:26.494 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.288 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:26.501 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.294 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:26.530 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.306 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:26.542 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.322 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:26.535 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.337+0000 7fe9063d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.337+0000 7fe9063d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.337+0000 7fe9063d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:10.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.337+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe90013d7c0 con 0x7fe9001430c0 2024-03-25T18:35:10.346 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:26.582 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:10.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.374+0000 7fe8fcff9640 1 -- 172.21.15.102:0/3549978322 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fe8f8052560 con 0x7fe9001430c0 2024-03-25T18:35:10.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.374+0000 7fe8fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:10.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.374+0000 7fe8fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:10.375 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:35:10.375 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 10 librados: watch_flush enter 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 10 librados: watch_flush exit 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 10 client.4622.objecter successfully canceled tick 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 >> v1:172.21.15.146:6800/3848252714 conn(0x7fe8ec06df10 legacy=0x7fe8ec0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 10 monclient: shutdown 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 >> v1:172.21.15.102:6790/0 conn(0x7fe9001430c0 legacy=0x7fe9001e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 shutdown_connections 2024-03-25T18:35:10.378 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 >> 172.21.15.102:0/3549978322 conn(0x7fe9000b7ea0 msgr2=0x7fe900074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:10.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 shutdown_connections 2024-03-25T18:35:10.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 1 -- 172.21.15.102:0/3549978322 wait complete. 2024-03-25T18:35:10.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:10.376+0000 7fe9063d6640 1 librados: shutdown 2024-03-25T18:35:10.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:17.194 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.309+0000 7f292cf3b640 10 monclient: get_monmap_and_config 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.309+0000 7f292cf3b640 10 monclient: build_initial_monmap 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.309+0000 7f292cf3b640 10 monclient: monmap: 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:35:17.310405+0000 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:35:17.310405+0000 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:17.310 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:17.311 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:17.311 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:35:17.311 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:35:17.311 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:17.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.310+0000 7f292cf3b640 1 Processor -- start 2024-03-25T18:35:17.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.310+0000 7f292cf3b640 1 -- start start 2024-03-25T18:35:17.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.310+0000 7f292cf3b640 10 monclient: init 2024-03-25T18:35:17.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:17.312 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:17.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 10 monclient(hunting): picked mon.noname-a con 0x7f29280b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:17.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 10 monclient(hunting): picked mon.noname-c con 0x7f29280b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:17.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 10 monclient(hunting): picked mon.noname-b con 0x7f292814be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:17.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280af940 con 0x7f29280b4d60 2024-03-25T18:35:17.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280b0a70 con 0x7f29280b1730 2024-03-25T18:35:17.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280b0d50 con 0x7f292814be90 2024-03-25T18:35:17.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:17.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.311+0000 7f292cf3b640 10 monclient(hunting): authenticate will time out at 7948.599272s 2024-03-25T18:35:17.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.321+0000 7f2925d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f29280b1730 0x7f29280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:36176/0 (socket says 172.21.15.102:36176) 2024-03-25T18:35:17.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.321+0000 7f2925d9b640 1 -- 172.21.15.102:0/2936485648 learned_addr learned my addr 172.21.15.102:0/2936485648 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:17.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.340+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513879333 0 0) 0x7f29280af940 con 0x7f29280b4d60 2024-03-25T18:35:17.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.340+0000 7f292559a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.340+0000 7f292559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.340+0000 7f292559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.340+0000 7f292559a640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:35:17.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.340+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2910003520 con 0x7f29280b4d60 2024-03-25T18:35:17.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.342+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535857186 0 0) 0x7f29280b0a70 con 0x7f29280b1730 2024-03-25T18:35:17.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.342+0000 7f292559a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.342+0000 7f292559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.342+0000 7f292559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.342+0000 7f292559a640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:35:17.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.342+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29280af940 con 0x7f29280b1730 2024-03-25T18:35:17.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.346+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508562614 0 0) 0x7f29280b0d50 con 0x7f292814be90 2024-03-25T18:35:17.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.346+0000 7f292559a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.346+0000 7f292559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.346+0000 7f292559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.346+0000 7f292559a640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:35:17.347 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.346+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29280b0a70 con 0x7f292814be90 2024-03-25T18:35:17.358 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859041081 0 0) 0x7f2910003520 con 0x7f29280b4d60 2024-03-25T18:35:17.358 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 >> v1:172.21.15.102:6790/0 conn(0x7f292814be90 legacy=0x7f292814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.358 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 >> v1:172.21.15.146:6789/0 conn(0x7f29280b1730 legacy=0x7f29280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 10 monclient: _finish_hunting 0 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 1 monclient: found mon.noname-a 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280b1030 con 0x7f29280b4d60 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 10 monclient: _finish_auth 0 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 10 monclient: _check_auth_tickets 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f29180015c0 con 0x7f29280b4d60 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292cf3b640 5 monclient: authenticate success, global_id 4641 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292cf3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292559a640 10 monclient: dump: 2024-03-25T18:35:17.359 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:17.360 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.357+0000 7f292cf3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.379+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2918001720 con 0x7f29280b4d60 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.379+0000 7f292559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.379+0000 7f292559a640 1 -- 172.21.15.102:0/2936485648 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2918001b30 con 0x7f29280b4d60 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.379+0000 7f292559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.379+0000 7f292559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.379+0000 7f292559a640 10 monclient: dump: 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:17.380 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.379+0000 7f292cf3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:17.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.379+0000 7f292cf3b640 4 set_mon_vals no callback set 2024-03-25T18:35:17.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.380+0000 7f292cf3b640 10 monclient: shutdown 2024-03-25T18:35:17.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.380+0000 7f292cf3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:17.383 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.380+0000 7f292cf3b640 1 -- 172.21.15.102:0/2936485648 >> v1:172.21.15.102:6789/0 conn(0x7f29280b4d60 legacy=0x7f292813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.380+0000 7f292cf3b640 1 -- 172.21.15.102:0/2936485648 shutdown_connections 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.380+0000 7f292cf3b640 1 -- 172.21.15.102:0/2936485648 >> 172.21.15.102:0/2936485648 conn(0x7f29280617a0 msgr2=0x7f2928061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.380+0000 7f292cf3b640 1 -- 172.21.15.102:0/2936485648 shutdown_connections 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.380+0000 7f292cf3b640 1 -- 172.21.15.102:0/2936485648 wait complete. 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.381+0000 7f292cf3b640 10 monclient: build_initial_monmap 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.381+0000 7f292cf3b640 10 monclient: monmap: 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:35:17.384 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.381+0000 7f292cf3b640 1 librados: starting msgr at 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.381+0000 7f292cf3b640 1 librados: starting objecter 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 1 Processor -- start 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 1 -- start start 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 1 librados: setting wanted keys 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 1 librados: calling monclient init 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 10 monclient: init 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 10 monclient(hunting): picked mon.noname-c con 0x7f29280b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 10 monclient(hunting): picked mon.noname-b con 0x7f29280b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 10 monclient(hunting): picked mon.noname-a con 0x7f292814be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2928138330 con 0x7f292814be90 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29281361b0 con 0x7f29280b4d60 2024-03-25T18:35:17.385 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29281385a0 con 0x7f29280b1730 2024-03-25T18:35:17.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:17.386 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.382+0000 7f292cf3b640 10 monclient(hunting): authenticate will time out at 7948.670866s 2024-03-25T18:35:17.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.386+0000 7f2925d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f29280b4d60 0x7f29281393b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:36248/0 (socket says 172.21.15.102:36248) 2024-03-25T18:35:17.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.386+0000 7f2925d9b640 1 -- 172.21.15.102:0/4008218930 learned_addr learned my addr 172.21.15.102:0/4008218930 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:17.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.386+0000 7f2916ffd640 10 client.?.objecter ms_handle_connect 0x7f29280b4d60 2024-03-25T18:35:17.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.386+0000 7f2916ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.391+0000 7f2916ffd640 10 client.?.objecter ms_handle_connect 0x7f29280b1730 2024-03-25T18:35:17.392 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.391+0000 7f2916ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614536368 0 0) 0x7f29281385a0 con 0x7f29280b1730 2024-03-25T18:35:17.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2908004eb0 con 0x7f29280b1730 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739668796 0 0) 0x7f29281361b0 con 0x7f29280b4d60 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:35:17.400 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.399+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29281385a0 con 0x7f29280b4d60 2024-03-25T18:35:17.408 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.408+0000 7f2916ffd640 10 client.?.objecter ms_handle_connect 0x7f292814be90 2024-03-25T18:35:17.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.408+0000 7f2916ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:17.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.409+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895917510 0 0) 0x7f29281385a0 con 0x7f29280b4d60 2024-03-25T18:35:17.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.409+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29281361b0 con 0x7f29280b4d60 2024-03-25T18:35:17.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.409+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f291c0019b0 con 0x7f29280b4d60 2024-03-25T18:35:17.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.409+0000 7f2916ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.409+0000 7f2916ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:17.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.409+0000 7f2916ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:17.410 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:17.410 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:17.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.409+0000 7f2916ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.411+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2608268678 0 0) 0x7f2908004eb0 con 0x7f29280b1730 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.411+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29281385a0 con 0x7f29280b1730 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.411+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2918001a30 con 0x7f29280b1730 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.411+0000 7f2916ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.411+0000 7f2916ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.411+0000 7f2916ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.412 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:17.413 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:17.413 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:17.413 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:17.413 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2876826375 0 0) 0x7f29281385a0 con 0x7f29280b1730 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 >> v1:172.21.15.146:6789/0 conn(0x7f29280b4d60 legacy=0x7f29281393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 >> v1:172.21.15.102:6789/0 conn(0x7f292814be90 legacy=0x7f292812fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 1 monclient: found mon.c 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292813b450 con 0x7f29280b1730 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f2916ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:17.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:35:17.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 10 monclient: _renew_subs 2024-03-25T18:35:17.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:17.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2928144dd0 con 0x7f29280b1730 2024-03-25T18:35:17.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:17.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 10 monclient: _renew_subs 2024-03-25T18:35:17.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:17.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2928130770 con 0x7f29280b1730 2024-03-25T18:35:17.421 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.419+0000 7f292cf3b640 1 librados: init done 2024-03-25T18:35:17.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.420+0000 7f292cf3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:17.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.420+0000 7f292cf3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:17.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.420+0000 7f292cf3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:17.422 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.420+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29280627b0 con 0x7f29280b1730 2024-03-25T18:35:17.725 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.724+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29180038a0 con 0x7f29280b1730 2024-03-25T18:35:17.725 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.724+0000 7f2916ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:17.725 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.724+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2918003bb0 con 0x7f29280b1730 2024-03-25T18:35:17.725 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.724+0000 7f2916ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.724+0000 7f2916ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.724+0000 7f2916ffd640 10 monclient: dump: 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:17.726 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.724+0000 7f2924d99640 4 set_mon_vals no callback set 2024-03-25T18:35:17.727 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.725+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f2918019b80 con 0x7f29280b1730 2024-03-25T18:35:17.727 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.726+0000 7f2925d9b640 10 monclient: get_auth_request con 0x7f290806df10 auth_method 0 2024-03-25T18:35:17.727 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.726+0000 7f2916ffd640 10 client.4628.objecter ms_handle_connect 0x7f290806df10 2024-03-25T18:35:17.732 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.732+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f2918053660 con 0x7f29280b1730 2024-03-25T18:35:17.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.732+0000 7f2916ffd640 10 client.4628.objecter ms_dispatch 0x7f2928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:17.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.732+0000 7f2916ffd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:17.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.732+0000 7f2916ffd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:17.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.732+0000 7f2916ffd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T18:35:17.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.732+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f291804f170 con 0x7f29280b1730 2024-03-25T18:35:17.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.732+0000 7f2916ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:17.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.732+0000 7f2916ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:17.983 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.983+0000 7f292cf3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:17.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.983+0000 7f292cf3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:17.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.983+0000 7f292cf3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:35:17.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:17.983+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29280b3c30 con 0x7f29280b1730 2024-03-25T18:35:18.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.004+0000 7f2916ffd640 1 -- 172.21.15.102:0/4008218930 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f29180510a0 con 0x7f29280b1730 2024-03-25T18:35:18.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.004+0000 7f2916ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:18.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.004+0000 7f2916ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:18.005 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:35:18.005 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 10 librados: watch_flush enter 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 10 librados: watch_flush exit 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 10 client.4628.objecter successfully canceled tick 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 >> v1:172.21.15.146:6800/3848252714 conn(0x7f290806df10 legacy=0x7f29080703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 10 monclient: shutdown 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 >> v1:172.21.15.102:6790/0 conn(0x7f29280b1730 legacy=0x7f2928137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 shutdown_connections 2024-03-25T18:35:18.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 >> 172.21.15.102:0/4008218930 conn(0x7f29280617a0 msgr2=0x7f292813c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:18.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 shutdown_connections 2024-03-25T18:35:18.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 1 -- 172.21.15.102:0/4008218930 wait complete. 2024-03-25T18:35:18.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:18.007+0000 7f292cf3b640 1 librados: shutdown 2024-03-25T18:35:18.020 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:24.827 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.943+0000 7fd84ec64640 10 monclient: get_monmap_and_config 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.943+0000 7fd84ec64640 10 monclient: build_initial_monmap 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.943+0000 7fd84ec64640 10 monclient: monmap: 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:35:24.943880+0000 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:35:24.943880+0000 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:35:24.944 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:35:24.945 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.944+0000 7fd84ec64640 1 Processor -- start 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.944+0000 7fd84ec64640 1 -- start start 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.944+0000 7fd84ec64640 10 monclient: init 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 10 monclient(hunting): picked mon.noname-a con 0x7fd8480a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 10 monclient(hunting): picked mon.noname-c con 0x7fd84813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 10 monclient(hunting): picked mon.noname-b con 0x7fd8481430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:24.946 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd848142520 con 0x7fd8480a98f0 2024-03-25T18:35:24.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8480b28a0 con 0x7fd84813f710 2024-03-25T18:35:24.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8480b2a30 con 0x7fd8481430c0 2024-03-25T18:35:24.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:24.947 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.945+0000 7fd84ec64640 10 monclient(hunting): authenticate will time out at 7956.233496s 2024-03-25T18:35:24.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.950+0000 7fd847fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd84813f710 0x7fd8481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:44294/0 (socket says 172.21.15.102:44294) 2024-03-25T18:35:24.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.950+0000 7fd847fff640 1 -- 172.21.15.102:0/3691448524 learned_addr learned my addr 172.21.15.102:0/3691448524 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:24.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.968+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422224500 0 0) 0x7fd848142520 con 0x7fd8480a98f0 2024-03-25T18:35:24.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.968+0000 7fd8477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:24.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.968+0000 7fd8477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:24.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.968+0000 7fd8477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:24.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.968+0000 7fd8477fe640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:35:24.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.968+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd838003520 con 0x7fd8480a98f0 2024-03-25T18:35:24.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.972+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897550073 0 0) 0x7fd8480b28a0 con 0x7fd84813f710 2024-03-25T18:35:24.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.972+0000 7fd8477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:24.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.972+0000 7fd8477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:24.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.972+0000 7fd8477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:24.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.972+0000 7fd8477fe640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:35:24.973 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.972+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd848142520 con 0x7fd84813f710 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 972135648 0 0) 0x7fd838003520 con 0x7fd8480a98f0 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 >> v1:172.21.15.102:6790/0 conn(0x7fd8481430c0 legacy=0x7fd84814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 >> v1:172.21.15.146:6789/0 conn(0x7fd84813f710 legacy=0x7fd8481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 1 monclient: found mon.noname-a 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8480b44c0 con 0x7fd8480a98f0 2024-03-25T18:35:24.991 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd83c001690 con 0x7fd8480a98f0 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd8477fe640 10 monclient: dump: 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:24.992 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:24.993 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:24.993 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:24.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd84ec64640 5 monclient: authenticate success, global_id 4647 2024-03-25T18:35:24.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.990+0000 7fd84ec64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.998+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd83c0017f0 con 0x7fd8480a98f0 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.998+0000 7fd8477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.998+0000 7fd8477fe640 1 -- 172.21.15.102:0/3691448524 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd83c001c00 con 0x7fd8480a98f0 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.998+0000 7fd8477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.998+0000 7fd8477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.998+0000 7fd8477fe640 10 monclient: dump: 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:24.999 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.000 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.000 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:25.000 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:25.000 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:25.000 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:25.000 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:25.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.998+0000 7fd84ec64640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:25.000 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.998+0000 7fd84ec64640 4 set_mon_vals no callback set 2024-03-25T18:35:25.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.999+0000 7fd84ec64640 10 monclient: shutdown 2024-03-25T18:35:25.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.999+0000 7fd84ec64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.999+0000 7fd84ec64640 1 -- 172.21.15.102:0/3691448524 >> v1:172.21.15.102:6789/0 conn(0x7fd8480a98f0 legacy=0x7fd8480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.999+0000 7fd84ec64640 1 -- 172.21.15.102:0/3691448524 shutdown_connections 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.999+0000 7fd84ec64640 1 -- 172.21.15.102:0/3691448524 >> 172.21.15.102:0/3691448524 conn(0x7fd8480b7ea0 msgr2=0x7fd8480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.999+0000 7fd84ec64640 1 -- 172.21.15.102:0/3691448524 shutdown_connections 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:24.999+0000 7fd84ec64640 1 -- 172.21.15.102:0/3691448524 wait complete. 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 10 monclient: build_initial_monmap 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 10 monclient: monmap: 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:35:25.003 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 1 librados: starting msgr at 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 1 librados: starting objecter 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 1 Processor -- start 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 1 -- start start 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 1 librados: setting wanted keys 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 1 librados: calling monclient init 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.000+0000 7fd84ec64640 10 monclient: init 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 10 monclient(hunting): picked mon.noname-a con 0x7fd8480a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:25.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 10 monclient(hunting): picked mon.noname-c con 0x7fd84813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 10 monclient(hunting): picked mon.noname-b con 0x7fd8481430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd848136bf0 con 0x7fd8480a98f0 2024-03-25T18:35:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd848137420 con 0x7fd8481430c0 2024-03-25T18:35:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd848136e60 con 0x7fd84813f710 2024-03-25T18:35:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:25.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.001+0000 7fd84ec64640 10 monclient(hunting): authenticate will time out at 7956.289696s 2024-03-25T18:35:25.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.012+0000 7fd84d201640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd8481430c0 0x7fd8481e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:44344/0 (socket says 172.21.15.102:44344) 2024-03-25T18:35:25.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.012+0000 7fd84d201640 1 -- 172.21.15.102:0/2263796928 learned_addr learned my addr 172.21.15.102:0/2263796928 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:25.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.015+0000 7fd8457fa640 10 client.?.objecter ms_handle_connect 0x7fd8481430c0 2024-03-25T18:35:25.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.015+0000 7fd8457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:25.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.025+0000 7fd8457fa640 10 client.?.objecter ms_handle_connect 0x7fd84813f710 2024-03-25T18:35:25.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.025+0000 7fd8457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:25.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.027+0000 7fd8457fa640 10 client.?.objecter ms_handle_connect 0x7fd8480a98f0 2024-03-25T18:35:25.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.027+0000 7fd8457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:25.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.032+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500373450 0 0) 0x7fd848137420 con 0x7fd8481430c0 2024-03-25T18:35:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.032+0000 7fd8457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.032+0000 7fd8457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.032+0000 7fd8457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.032+0000 7fd8457fa640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:35:25.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.032+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd82c004eb0 con 0x7fd8481430c0 2024-03-25T18:35:25.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790268615 0 0) 0x7fd848136e60 con 0x7fd84813f710 2024-03-25T18:35:25.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:35:25.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd848137420 con 0x7fd84813f710 2024-03-25T18:35:25.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024899888 0 0) 0x7fd848136bf0 con 0x7fd8480a98f0 2024-03-25T18:35:25.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 10 monclient(hunting): _negotiate 2024-03-25T18:35:25.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:25.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:25.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:35:25.043 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.041+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd848136e60 con 0x7fd8480a98f0 2024-03-25T18:35:25.059 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.058+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578169178 0 0) 0x7fd82c004eb0 con 0x7fd8481430c0 2024-03-25T18:35:25.059 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.058+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd848136bf0 con 0x7fd8481430c0 2024-03-25T18:35:25.059 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.058+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd840001c90 con 0x7fd8481430c0 2024-03-25T18:35:25.059 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.058+0000 7fd8457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.059 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.058+0000 7fd8457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:25.059 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.058+0000 7fd8457fa640 10 monclient(hunting): dump: 2024-03-25T18:35:25.059 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:25.059 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:25.060 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.058+0000 7fd8457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:25.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.069+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 989888549 0 0) 0x7fd848136e60 con 0x7fd8480a98f0 2024-03-25T18:35:25.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.069+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd82c004eb0 con 0x7fd8480a98f0 2024-03-25T18:35:25.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.069+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd83c001b00 con 0x7fd8480a98f0 2024-03-25T18:35:25.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.069+0000 7fd8457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.069+0000 7fd8457fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.069+0000 7fd8457fa640 10 monclient(hunting): dump: 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:25.071 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 417235274 0 0) 0x7fd848137420 con 0x7fd84813f710 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd848136e60 con 0x7fd84813f710 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd834001930 con 0x7fd84813f710 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 10 monclient(hunting): dump: 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:25.081 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2854104632 0 0) 0x7fd848136bf0 con 0x7fd8481430c0 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 >> v1:172.21.15.102:6790/0 conn(0x7fd84813f710 legacy=0x7fd8481355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 >> v1:172.21.15.102:6789/0 conn(0x7fd8480a98f0 legacy=0x7fd848137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 10 monclient: _finish_hunting 0 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 1 monclient: found mon.b 2024-03-25T18:35:25.082 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd848137140 con 0x7fd8481430c0 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 10 monclient: _finish_auth 0 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 10 monclient: _check_auth_tickets 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd8457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 10 monclient: _renew_subs 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8480b7990 con 0x7fd8481430c0 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 10 monclient: _renew_subs 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84813b990 con 0x7fd8481430c0 2024-03-25T18:35:25.083 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.079+0000 7fd84ec64640 1 librados: init done 2024-03-25T18:35:25.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.081+0000 7fd84ec64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:25.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.081+0000 7fd84ec64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:25.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.081+0000 7fd84ec64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:35:25.084 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.081+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd848074cc0 con 0x7fd8481430c0 2024-03-25T18:35:25.092 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.092+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd840003990 con 0x7fd8481430c0 2024-03-25T18:35:25.093 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.092+0000 7fd8457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:25.093 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.092+0000 7fd846ffd640 4 set_mon_vals no callback set 2024-03-25T18:35:25.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.361+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fd840003ce0 con 0x7fd8481430c0 2024-03-25T18:35:25.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.361+0000 7fd8457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:25.362 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.361+0000 7fd8457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.361+0000 7fd8457fa640 10 monclient: dump: 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:25.363 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:25.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.362+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fd840003ec0 con 0x7fd8481430c0 2024-03-25T18:35:25.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.363+0000 7fd84ca00640 10 monclient: get_auth_request con 0x7fd82c06e0b0 auth_method 0 2024-03-25T18:35:25.364 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.363+0000 7fd8457fa640 10 client.4657.objecter ms_handle_connect 0x7fd82c06e0b0 2024-03-25T18:35:25.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.365+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fd840054b60 con 0x7fd8481430c0 2024-03-25T18:35:25.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.365+0000 7fd8457fa640 10 client.4657.objecter ms_dispatch 0x7fd848000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:25.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.365+0000 7fd8457fa640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:25.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.365+0000 7fd8457fa640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:25.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.365+0000 7fd8457fa640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T18:35:25.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.368+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd840050670 con 0x7fd8481430c0 2024-03-25T18:35:25.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.368+0000 7fd8457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:25.369 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.368+0000 7fd8457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:25.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.620+0000 7fd84ec64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:25.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.620+0000 7fd84ec64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:25.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.620+0000 7fd84ec64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:35:25.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.620+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8480627b0 con 0x7fd8481430c0 2024-03-25T18:35:25.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.653+0000 7fd8457fa640 1 -- 172.21.15.102:0/2263796928 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fd8400525a0 con 0x7fd8481430c0 2024-03-25T18:35:25.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.653+0000 7fd8457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:25.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.653+0000 7fd8457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:25.655 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:35:25.655 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:25.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.655+0000 7fd84ec64640 10 librados: watch_flush enter 2024-03-25T18:35:25.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 10 librados: watch_flush exit 2024-03-25T18:35:25.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 10 client.4657.objecter successfully canceled tick 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 >> v1:172.21.15.146:6800/3848252714 conn(0x7fd82c06e0b0 legacy=0x7fd82c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 10 monclient: shutdown 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 >> v1:172.21.15.146:6789/0 conn(0x7fd8481430c0 legacy=0x7fd8481e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 shutdown_connections 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 >> 172.21.15.102:0/2263796928 conn(0x7fd8480b7ea0 msgr2=0x7fd848074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 shutdown_connections 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 1 -- 172.21.15.102:0/2263796928 wait complete. 2024-03-25T18:35:25.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:25.656+0000 7fd84ec64640 1 librados: shutdown 2024-03-25T18:35:25.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:29.956 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:46.191 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:29.963 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:46.199 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:30.010 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:46.223 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:30.016 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:09:46.252 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:30.019 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:46.233 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:30.023 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:46.259 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:30.028 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:46.264 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:30.053 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:46.288 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:30.055 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:09:46.268 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:32.476 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:32.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.593+0000 7f386ce57640 10 monclient: get_monmap_and_config 2024-03-25T18:35:32.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.593+0000 7f386ce57640 10 monclient: build_initial_monmap 2024-03-25T18:35:32.595 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.593+0000 7f386ce57640 10 monclient: monmap: 2024-03-25T18:35:32.595 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:32.595 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:32.595 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:35:32.595001+0000 2024-03-25T18:35:32.596 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:35:32.595001+0000 2024-03-25T18:35:32.596 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:32.596 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:32.596 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:32.596 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:35:32.596 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:35:32.596 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:32.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.594+0000 7f386ce57640 1 Processor -- start 2024-03-25T18:35:32.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.594+0000 7f386ce57640 1 -- start start 2024-03-25T18:35:32.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.594+0000 7f386ce57640 10 monclient: init 2024-03-25T18:35:32.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:32.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:32.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 10 monclient(hunting): picked mon.noname-c con 0x7f386806e5c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:32.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 10 monclient(hunting): picked mon.noname-b con 0x7f3868071620 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:32.597 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 10 monclient(hunting): picked mon.noname-a con 0x7f386813ef90 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:32.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3868070530 con 0x7f386813ef90 2024-03-25T18:35:32.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3868070810 con 0x7f386806e5c0 2024-03-25T18:35:32.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3868070af0 con 0x7f3868071620 2024-03-25T18:35:32.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:32.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.595+0000 7f386ce57640 10 monclient(hunting): authenticate will time out at 7963.884302s 2024-03-25T18:35:32.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.601+0000 7f3866d9d640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f386813ef90 0x7f3868149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51294/0 (socket says 172.21.15.102:51294) 2024-03-25T18:35:32.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.601+0000 7f3866d9d640 1 -- 172.21.15.102:0/2024039147 learned_addr learned my addr 172.21.15.102:0/2024039147 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:32.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.620+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34938032 0 0) 0x7f3868070530 con 0x7f386813ef90 2024-03-25T18:35:32.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.620+0000 7f386559a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.620+0000 7f386559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.620+0000 7f386559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.620+0000 7f386559a640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:35:32.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.620+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3850004630 con 0x7f386813ef90 2024-03-25T18:35:32.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.621+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232170109 0 0) 0x7f3868070af0 con 0x7f3868071620 2024-03-25T18:35:32.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.621+0000 7f386559a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.621+0000 7f386559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.621+0000 7f386559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.621+0000 7f386559a640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:35:32.623 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.621+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3868070530 con 0x7f3868071620 2024-03-25T18:35:32.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.623+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220490898 0 0) 0x7f3868070810 con 0x7f386806e5c0 2024-03-25T18:35:32.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.623+0000 7f386559a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.623+0000 7f386559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.623+0000 7f386559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.623+0000 7f386559a640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:35:32.625 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.623+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3868070af0 con 0x7f386806e5c0 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1150105185 0 0) 0x7f3868070530 con 0x7f3868071620 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 >> v1:172.21.15.146:6789/0 conn(0x7f386806e5c0 legacy=0x7f3868070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 >> v1:172.21.15.102:6789/0 conn(0x7f386813ef90 legacy=0x7f3868149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 10 monclient: _finish_hunting 0 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 1 monclient: found mon.noname-b 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f386806f3f0 con 0x7f3868071620 2024-03-25T18:35:32.631 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 10 monclient: _finish_auth 0 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 10 monclient: _check_auth_tickets 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f385c001670 con 0x7f3868071620 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386ce57640 5 monclient: authenticate success, global_id 4637 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386ce57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386559a640 10 monclient: dump: 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.632 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:32.633 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:32.633 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:32.633 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:32.633 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:32.633 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.629+0000 7f386ce57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:32.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.635+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f385c0017d0 con 0x7f3868071620 2024-03-25T18:35:32.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.635+0000 7f386559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:32.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.635+0000 7f386559a640 1 -- 172.21.15.102:0/2024039147 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f385c001be0 con 0x7f3868071620 2024-03-25T18:35:32.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.635+0000 7f386559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.635+0000 7f386559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.635+0000 7f386559a640 10 monclient: dump: 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.635+0000 7f386ce57640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:32.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.635+0000 7f386ce57640 4 set_mon_vals no callback set 2024-03-25T18:35:32.640 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.636+0000 7f386ce57640 10 monclient: shutdown 2024-03-25T18:35:32.640 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.636+0000 7f386ce57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:32.640 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.636+0000 7f386ce57640 1 -- 172.21.15.102:0/2024039147 >> v1:172.21.15.102:6790/0 conn(0x7f3868071620 legacy=0x7f386813e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.640 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.636+0000 7f386ce57640 1 -- 172.21.15.102:0/2024039147 shutdown_connections 2024-03-25T18:35:32.640 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.636+0000 7f386ce57640 1 -- 172.21.15.102:0/2024039147 >> 172.21.15.102:0/2024039147 conn(0x7f386806b630 msgr2=0x7f38680b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:32.640 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.636+0000 7f386ce57640 1 -- 172.21.15.102:0/2024039147 shutdown_connections 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.636+0000 7f386ce57640 1 -- 172.21.15.102:0/2024039147 wait complete. 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.637+0000 7f386ce57640 10 monclient: build_initial_monmap 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.637+0000 7f386ce57640 10 monclient: monmap: 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 1 librados: starting msgr at 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 1 librados: starting objecter 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 1 Processor -- start 2024-03-25T18:35:32.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 1 -- start start 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 1 librados: setting wanted keys 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 1 librados: calling monclient init 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 10 monclient: init 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.638+0000 7f386ce57640 10 monclient(hunting): picked mon.noname-b con 0x7f386806e5c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.639+0000 7f386ce57640 10 monclient(hunting): picked mon.noname-c con 0x7f3868071620 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.639+0000 7f386ce57640 10 monclient(hunting): picked mon.noname-a con 0x7f386813ef90 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.639+0000 7f386ce57640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38681dadc0 con 0x7f386813ef90 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.639+0000 7f386ce57640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38681c7a60 con 0x7f386806e5c0 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.639+0000 7f386ce57640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38681c68b0 con 0x7f3868071620 2024-03-25T18:35:32.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.639+0000 7f386ce57640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:32.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.639+0000 7f386ce57640 10 monclient(hunting): authenticate will time out at 7963.928175s 2024-03-25T18:35:32.651 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.649+0000 7f3866d9d640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f386813ef90 0x7f386813bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51324/0 (socket says 172.21.15.102:51324) 2024-03-25T18:35:32.651 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.649+0000 7f3866d9d640 1 -- 172.21.15.102:0/3360103357 learned_addr learned my addr 172.21.15.102:0/3360103357 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:32.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.654+0000 7f3856ffd640 10 client.?.objecter ms_handle_connect 0x7f386806e5c0 2024-03-25T18:35:32.656 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.654+0000 7f3856ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:32.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.655+0000 7f3856ffd640 10 client.?.objecter ms_handle_connect 0x7f386813ef90 2024-03-25T18:35:32.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.655+0000 7f3856ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:32.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.656+0000 7f3856ffd640 10 client.?.objecter ms_handle_connect 0x7f3868071620 2024-03-25T18:35:32.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.656+0000 7f3856ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:32.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.671+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320719371 0 0) 0x7f38681dadc0 con 0x7f386813ef90 2024-03-25T18:35:32.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.671+0000 7f3856ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.671+0000 7f3856ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.671+0000 7f3856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.671+0000 7f3856ffd640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:35:32.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.671+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3848004eb0 con 0x7f386813ef90 2024-03-25T18:35:32.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.675+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795658352 0 0) 0x7f38681c7a60 con 0x7f386806e5c0 2024-03-25T18:35:32.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.675+0000 7f3856ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.675+0000 7f3856ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.675+0000 7f3856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.675+0000 7f3856ffd640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:35:32.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.675+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38681dadc0 con 0x7f386806e5c0 2024-03-25T18:35:32.685 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.683+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322041657 0 0) 0x7f38681c68b0 con 0x7f3868071620 2024-03-25T18:35:32.685 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.683+0000 7f3856ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:32.685 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.683+0000 7f3856ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:32.685 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.683+0000 7f3856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:32.685 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.683+0000 7f3856ffd640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:35:32.685 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.683+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38681c7a60 con 0x7f3868071620 2024-03-25T18:35:32.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.687+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477311051 0 0) 0x7f38681dadc0 con 0x7f386806e5c0 2024-03-25T18:35:32.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.687+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38681c68b0 con 0x7f386806e5c0 2024-03-25T18:35:32.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.687+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3858001870 con 0x7f386806e5c0 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.687+0000 7f3856ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.687+0000 7f3856ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.687+0000 7f3856ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:32.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.687+0000 7f3856ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:32.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.691+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226584066 0 0) 0x7f3848004eb0 con 0x7f386813ef90 2024-03-25T18:35:32.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38681dadc0 con 0x7f386813ef90 2024-03-25T18:35:32.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3860001b30 con 0x7f386813ef90 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:32.694 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1286711841 0 0) 0x7f38681c7a60 con 0x7f3868071620 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3848004eb0 con 0x7f3868071620 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f385c001ae0 con 0x7f3868071620 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.692+0000 7f3856ffd640 10 monclient(hunting): dump: 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:32.695 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.696 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:32.696 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:32.696 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:32.696 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:32.696 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:32.696 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:32.696 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:32.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1505076418 0 0) 0x7f38681dadc0 con 0x7f386813ef90 2024-03-25T18:35:32.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 >> v1:172.21.15.102:6790/0 conn(0x7f3868071620 legacy=0x7f38681db2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 >> v1:172.21.15.146:6789/0 conn(0x7f386806e5c0 legacy=0x7f38681c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:32.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:32.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 1 monclient: found mon.a 2024-03-25T18:35:32.708 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38681c6a40 con 0x7f386813ef90 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 10 monclient: _finish_auth 0 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f3856ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 5 monclient: authenticate success, global_id 4656 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 10 monclient: _renew_subs 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38681db9d0 con 0x7f386813ef90 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 10 client.4656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 10 monclient: _renew_subs 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38681c55e0 con 0x7f386813ef90 2024-03-25T18:35:32.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.706+0000 7f386ce57640 1 librados: init done 2024-03-25T18:35:32.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.707+0000 7f386ce57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:32.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.707+0000 7f386ce57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:32.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.708+0000 7f386ce57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:32.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:32.708+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f386806fd40 con 0x7f386813ef90 2024-03-25T18:35:33.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.028+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3860002040 con 0x7f386813ef90 2024-03-25T18:35:33.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.028+0000 7f3856ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:33.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.028+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f3860004d00 con 0x7f386813ef90 2024-03-25T18:35:33.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.028+0000 7f3856ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:33.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.028+0000 7f3856ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:33.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.028+0000 7f3856ffd640 10 monclient: dump: 2024-03-25T18:35:33.030 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:33.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.028+0000 7f3864d99640 4 set_mon_vals no callback set 2024-03-25T18:35:33.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.029+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f386001ad50 con 0x7f386813ef90 2024-03-25T18:35:33.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.030+0000 7f386659c640 10 monclient: get_auth_request con 0x7f384806e160 auth_method 0 2024-03-25T18:35:33.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.030+0000 7f3856ffd640 10 client.4656.objecter ms_handle_connect 0x7f384806e160 2024-03-25T18:35:33.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.034+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f3860054aa0 con 0x7f386813ef90 2024-03-25T18:35:33.036 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.034+0000 7f3856ffd640 10 client.4656.objecter ms_dispatch 0x7f3868000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:33.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.034+0000 7f3856ffd640 3 client.4656.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:33.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.034+0000 7f3856ffd640 3 client.4656.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:33.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.035+0000 7f3856ffd640 20 client.4656.objecter dump_active .. 0 homeless 2024-03-25T18:35:33.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.035+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38600503e0 con 0x7f386813ef90 2024-03-25T18:35:33.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.035+0000 7f3856ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:33.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.035+0000 7f3856ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:33.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.261+0000 7f386ce57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.261+0000 7f386ce57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.261+0000 7f386ce57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:33.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.261+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f386806dc80 con 0x7f386813ef90 2024-03-25T18:35:33.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.287+0000 7f3856ffd640 1 -- 172.21.15.102:0/3360103357 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f386806dc80 con 0x7f386813ef90 2024-03-25T18:35:33.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.287+0000 7f3856ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:33.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.287+0000 7f3856ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:33.289 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:35:33.289 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 10 librados: watch_flush enter 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 10 librados: watch_flush exit 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 20 client.4656.objecter shutdown clearing up homeless session... 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 10 client.4656.objecter successfully canceled tick 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 >> v1:172.21.15.146:6800/3848252714 conn(0x7f384806e160 legacy=0x7f3848070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 10 monclient: shutdown 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 >> v1:172.21.15.102:6789/0 conn(0x7f386813ef90 legacy=0x7f386813bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 shutdown_connections 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 >> 172.21.15.102:0/3360103357 conn(0x7f386806b630 msgr2=0x7f38680aed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 shutdown_connections 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 1 -- 172.21.15.102:0/3360103357 wait complete. 2024-03-25T18:35:33.292 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:33.290+0000 7f386ce57640 1 librados: shutdown 2024-03-25T18:35:33.303 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:40.110 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:40.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.225+0000 7fa68527d640 10 monclient: get_monmap_and_config 2024-03-25T18:35:40.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.225+0000 7fa68527d640 10 monclient: build_initial_monmap 2024-03-25T18:35:40.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.225+0000 7fa68527d640 10 monclient: monmap: 2024-03-25T18:35:40.227 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:40.227 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:40.227 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:35:40.227090+0000 2024-03-25T18:35:40.227 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:35:40.227090+0000 2024-03-25T18:35:40.228 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:40.228 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:40.228 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:40.228 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:35:40.228 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:35:40.228 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.226+0000 7fa68527d640 1 Processor -- start 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.226+0000 7fa68527d640 1 -- start start 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.226+0000 7fa68527d640 10 monclient: init 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.226+0000 7fa68527d640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6800b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6800b1730 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 10 monclient(hunting): picked mon.noname-b con 0x7fa68014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:40.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800af940 con 0x7fa6800b1730 2024-03-25T18:35:40.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800b0a70 con 0x7fa6800b4d60 2024-03-25T18:35:40.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800b0d50 con 0x7fa68014be90 2024-03-25T18:35:40.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:40.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa68527d640 10 monclient(hunting): authenticate will time out at 7971.516050s 2024-03-25T18:35:40.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa67e7fc640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa6800b1730 0x7fa6800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56296/0 (socket says 172.21.15.102:56296) 2024-03-25T18:35:40.231 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.227+0000 7fa67e7fc640 1 -- 172.21.15.102:0/2799419829 learned_addr learned my addr 172.21.15.102:0/2799419829 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:40.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.288+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417592143 0 0) 0x7fa6800b0d50 con 0x7fa68014be90 2024-03-25T18:35:40.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.288+0000 7fa67dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.288+0000 7fa67dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.288+0000 7fa67dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.288+0000 7fa67dffb640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:35:40.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.288+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa660003520 con 0x7fa68014be90 2024-03-25T18:35:40.314 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.312+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798263411 0 0) 0x7fa6800af940 con 0x7fa6800b1730 2024-03-25T18:35:40.314 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.312+0000 7fa67dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.314 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.312+0000 7fa67dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.314 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.312+0000 7fa67dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.314 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.312+0000 7fa67dffb640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:35:40.314 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.312+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6800b0d50 con 0x7fa6800b1730 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440559666 0 0) 0x7fa6800b0a70 con 0x7fa6800b4d60 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6800af940 con 0x7fa6800b4d60 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1620381121 0 0) 0x7fa660003520 con 0x7fa68014be90 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 >> v1:172.21.15.146:6789/0 conn(0x7fa6800b4d60 legacy=0x7fa68013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.321 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 >> v1:172.21.15.102:6789/0 conn(0x7fa6800b1730 legacy=0x7fa6800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 1 monclient: found mon.noname-b 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6800b1030 con 0x7fa68014be90 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient: _finish_auth 0 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa670001690 con 0x7fa68014be90 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa68527d640 5 monclient: authenticate success, global_id 4643 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa68527d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:40.322 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa67dffb640 10 monclient: dump: 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:40.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.318+0000 7fa68527d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:40.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.337+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6700017f0 con 0x7fa68014be90 2024-03-25T18:35:40.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.337+0000 7fa67dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:40.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.337+0000 7fa67dffb640 1 -- 172.21.15.102:0/2799419829 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa670001c00 con 0x7fa68014be90 2024-03-25T18:35:40.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.337+0000 7fa68527d640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:40.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.337+0000 7fa68527d640 4 set_mon_vals no callback set 2024-03-25T18:35:40.339 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.337+0000 7fa67dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.337+0000 7fa67dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.337+0000 7fa67dffb640 10 monclient: dump: 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:40.340 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:40.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.338+0000 7fa68527d640 10 monclient: shutdown 2024-03-25T18:35:40.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.338+0000 7fa68527d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:40.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.338+0000 7fa68527d640 1 -- 172.21.15.102:0/2799419829 >> v1:172.21.15.102:6790/0 conn(0x7fa68014be90 legacy=0x7fa68014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.338+0000 7fa68527d640 1 -- 172.21.15.102:0/2799419829 shutdown_connections 2024-03-25T18:35:40.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.338+0000 7fa68527d640 1 -- 172.21.15.102:0/2799419829 >> 172.21.15.102:0/2799419829 conn(0x7fa6800617a0 msgr2=0x7fa680061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:40.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.338+0000 7fa68527d640 1 -- 172.21.15.102:0/2799419829 shutdown_connections 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.338+0000 7fa68527d640 1 -- 172.21.15.102:0/2799419829 wait complete. 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.339+0000 7fa68527d640 10 monclient: build_initial_monmap 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.339+0000 7fa68527d640 10 monclient: monmap: 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:40.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.340+0000 7fa68527d640 1 librados: starting msgr at 2024-03-25T18:35:40.344 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.340+0000 7fa68527d640 1 librados: starting objecter 2024-03-25T18:35:40.344 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.340+0000 7fa68527d640 1 Processor -- start 2024-03-25T18:35:40.344 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.340+0000 7fa68527d640 1 -- start start 2024-03-25T18:35:40.344 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.340+0000 7fa68527d640 1 librados: setting wanted keys 2024-03-25T18:35:40.344 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.340+0000 7fa68527d640 1 librados: calling monclient init 2024-03-25T18:35:40.344 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.340+0000 7fa68527d640 10 monclient: init 2024-03-25T18:35:40.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:40.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:35:40.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6800b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:40.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6800b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:40.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 10 monclient(hunting): picked mon.noname-c con 0x7fa68014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:40.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6801378e0 con 0x7fa6800b4d60 2024-03-25T18:35:40.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6801409b0 con 0x7fa6800b1730 2024-03-25T18:35:40.345 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa680140b40 con 0x7fa68014be90 2024-03-25T18:35:40.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:40.346 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.341+0000 7fa68527d640 10 monclient(hunting): authenticate will time out at 7971.630757s 2024-03-25T18:35:40.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.351+0000 7fa67effd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa6800b1730 0x7fa680136fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:40884/0 (socket says 172.21.15.102:40884) 2024-03-25T18:35:40.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.351+0000 7fa67effd640 1 -- 172.21.15.102:0/2846499863 learned_addr learned my addr 172.21.15.102:0/2846499863 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:40.355 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.354+0000 7fa6777fe640 10 client.?.objecter ms_handle_connect 0x7fa6800b1730 2024-03-25T18:35:40.356 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.354+0000 7fa6777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.358 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.356+0000 7fa6777fe640 10 client.?.objecter ms_handle_connect 0x7fa6800b4d60 2024-03-25T18:35:40.358 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.356+0000 7fa6777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.374 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492587172 0 0) 0x7fa6801409b0 con 0x7fa6800b1730 2024-03-25T18:35:40.374 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.374 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa664004eb0 con 0x7fa6800b1730 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555107970 0 0) 0x7fa6801378e0 con 0x7fa6800b4d60 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.372+0000 7fa6777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.373+0000 7fa6777fe640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:35:40.375 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.373+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6801409b0 con 0x7fa6800b4d60 2024-03-25T18:35:40.376 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.375+0000 7fa6777fe640 10 client.?.objecter ms_handle_connect 0x7fa68014be90 2024-03-25T18:35:40.376 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.375+0000 7fa6777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:40.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.389+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509079188 0 0) 0x7fa680140b40 con 0x7fa68014be90 2024-03-25T18:35:40.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.389+0000 7fa6777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:35:40.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.389+0000 7fa6777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:40.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.389+0000 7fa6777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:40.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.389+0000 7fa6777fe640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:35:40.391 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.389+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6801378e0 con 0x7fa68014be90 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.396+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2407997477 0 0) 0x7fa6801409b0 con 0x7fa6800b4d60 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.396+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa680140b40 con 0x7fa6800b4d60 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.396+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa668001cf0 con 0x7fa6800b4d60 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.396+0000 7fa6777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.396+0000 7fa6777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.396+0000 7fa6777fe640 10 monclient(hunting): dump: 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.398 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.399 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:40.399 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:40.399 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:40.399 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:40.399 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:40.399 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.396+0000 7fa6777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.402+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3009432253 0 0) 0x7fa664004eb0 con 0x7fa6800b1730 2024-03-25T18:35:40.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.402+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6801409b0 con 0x7fa6800b1730 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.402+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa678001a10 con 0x7fa6800b1730 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.402+0000 7fa6777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.402+0000 7fa6777fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.402+0000 7fa6777fe640 10 monclient(hunting): dump: 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:40.405 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:40.406 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3794615103 0 0) 0x7fa680140b40 con 0x7fa6800b4d60 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 >> v1:172.21.15.102:6790/0 conn(0x7fa68014be90 legacy=0x7fa6801dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 >> v1:172.21.15.146:6789/0 conn(0x7fa6800b1730 legacy=0x7fa680136fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 1 monclient: found mon.a 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6801354e0 con 0x7fa6800b4d60 2024-03-25T18:35:40.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 10 monclient: _finish_auth 0 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa6777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 10 monclient: _renew_subs 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa680135730 con 0x7fa6800b4d60 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 10 client.4662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 10 monclient: _renew_subs 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa680144da0 con 0x7fa6800b4d60 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.406+0000 7fa68527d640 1 librados: init done 2024-03-25T18:35:40.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.407+0000 7fa68527d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:40.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.407+0000 7fa68527d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:40.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.407+0000 7fa68527d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:40.411 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.407+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6801376e0 con 0x7fa6800b4d60 2024-03-25T18:35:40.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.715+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa668004b60 con 0x7fa6800b4d60 2024-03-25T18:35:40.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.715+0000 7fa6777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:40.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.715+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa668004e70 con 0x7fa6800b4d60 2024-03-25T18:35:40.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.715+0000 7fa6777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:40.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.715+0000 7fa6777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:40.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.715+0000 7fa6777fe640 10 monclient: dump: 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:40.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.715+0000 7fa67d7fa640 4 set_mon_vals no callback set 2024-03-25T18:35:40.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.716+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fa66801ae80 con 0x7fa6800b4d60 2024-03-25T18:35:40.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.717+0000 7fa67effd640 10 monclient: get_auth_request con 0x7fa66406df20 auth_method 0 2024-03-25T18:35:40.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.717+0000 7fa6777fe640 10 client.4662.objecter ms_handle_connect 0x7fa66406df20 2024-03-25T18:35:40.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.722+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fa668054c00 con 0x7fa6800b4d60 2024-03-25T18:35:40.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.722+0000 7fa6777fe640 10 client.4662.objecter ms_dispatch 0x7fa680000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:40.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.722+0000 7fa6777fe640 3 client.4662.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:40.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.722+0000 7fa6777fe640 3 client.4662.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:40.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.722+0000 7fa6777fe640 20 client.4662.objecter dump_active .. 0 homeless 2024-03-25T18:35:40.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.722+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa668050540 con 0x7fa6800b4d60 2024-03-25T18:35:40.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.722+0000 7fa6777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:40.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.722+0000 7fa6777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:40.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.973+0000 7fa68527d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:40.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.973+0000 7fa68527d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:40.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.973+0000 7fa68527d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:40.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:40.973+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6800627b0 con 0x7fa6800b4d60 2024-03-25T18:35:41.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.008+0000 7fa6777fe640 1 -- 172.21.15.102:0/2846499863 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fa668052640 con 0x7fa6800b4d60 2024-03-25T18:35:41.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.008+0000 7fa6777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:41.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.008+0000 7fa6777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:41.010 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:35:41.010 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.010+0000 7fa68527d640 10 librados: watch_flush enter 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 10 librados: watch_flush exit 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 20 client.4662.objecter shutdown clearing up homeless session... 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 10 client.4662.objecter successfully canceled tick 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 >> v1:172.21.15.146:6800/3848252714 conn(0x7fa66406df20 legacy=0x7fa6640703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 10 monclient: shutdown 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 >> v1:172.21.15.102:6789/0 conn(0x7fa6800b4d60 legacy=0x7fa680134dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 shutdown_connections 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 >> 172.21.15.102:0/2846499863 conn(0x7fa6800617a0 msgr2=0x7fa6800b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:41.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 shutdown_connections 2024-03-25T18:35:41.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 1 -- 172.21.15.102:0/2846499863 wait complete. 2024-03-25T18:35:41.014 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:41.011+0000 7fa68527d640 1 librados: shutdown 2024-03-25T18:35:41.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:47.832 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:47.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.946+0000 7fb603d8c640 10 monclient: get_monmap_and_config 2024-03-25T18:35:47.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.946+0000 7fb603d8c640 10 monclient: build_initial_monmap 2024-03-25T18:35:47.948 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.946+0000 7fb603d8c640 10 monclient: monmap: 2024-03-25T18:35:47.948 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:47.948 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:47.949 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:35:47.948440+0000 2024-03-25T18:35:47.949 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:35:47.948440+0000 2024-03-25T18:35:47.949 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:47.949 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:47.949 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:47.949 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:35:47.949 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:35:47.949 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:47.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.947+0000 7fb603d8c640 1 Processor -- start 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.947+0000 7fb603d8c640 1 -- start start 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.947+0000 7fb603d8c640 10 monclient: init 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 10 monclient(hunting): picked mon.noname-a con 0x7fb5fc061830 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 10 monclient(hunting): picked mon.noname-c con 0x7fb5fc13c9f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 10 monclient(hunting): picked mon.noname-b con 0x7fb5fc072af0 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc071fc0 con 0x7fb5fc061830 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc0707a0 con 0x7fb5fc13c9f0 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc0ae9e0 con 0x7fb5fc072af0 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:47.951 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.948+0000 7fb603d8c640 10 monclient(hunting): authenticate will time out at 7979.237852s 2024-03-25T18:35:47.952 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.951+0000 7fb601327640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb5fc13c9f0 0x7fb5fc0723e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:44944/0 (socket says 172.21.15.102:44944) 2024-03-25T18:35:47.952 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.951+0000 7fb601327640 1 -- 172.21.15.102:0/3007018459 learned_addr learned my addr 172.21.15.102:0/3007018459 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:47.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.966+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491009068 0 0) 0x7fb5fc0707a0 con 0x7fb5fc13c9f0 2024-03-25T18:35:47.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.966+0000 7fb600b26640 10 monclient(hunting): _negotiate 2024-03-25T18:35:47.967 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.966+0000 7fb600b26640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:47.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.966+0000 7fb600b26640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:47.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.966+0000 7fb600b26640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:35:47.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.966+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e8005730 con 0x7fb5fc13c9f0 2024-03-25T18:35:47.981 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.980+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216337677 0 0) 0x7fb5fc0ae9e0 con 0x7fb5fc072af0 2024-03-25T18:35:47.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.980+0000 7fb600b26640 10 monclient(hunting): _negotiate 2024-03-25T18:35:47.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.980+0000 7fb600b26640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:47.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.980+0000 7fb600b26640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:47.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.980+0000 7fb600b26640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:35:47.982 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.980+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5fc0707a0 con 0x7fb5fc072af0 2024-03-25T18:35:47.984 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2596977148 0 0) 0x7fb5e8005730 con 0x7fb5fc13c9f0 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 >> v1:172.21.15.102:6790/0 conn(0x7fb5fc072af0 legacy=0x7fb5fc14a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 >> v1:172.21.15.102:6789/0 conn(0x7fb5fc061830 legacy=0x7fb5fc13c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 10 monclient: _finish_hunting 0 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 1 monclient: found mon.noname-c 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc070cc0 con 0x7fb5fc13c9f0 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 10 monclient: _finish_auth 0 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 10 monclient: _check_auth_tickets 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb5ec001690 con 0x7fb5fc13c9f0 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb603d8c640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb603d8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:47.985 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb600b26640 10 monclient: dump: 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:47.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:47.983+0000 7fb603d8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.006+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5ec001870 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.006+0000 7fb600b26640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.006+0000 7fb600b26640 1 -- 172.21.15.102:0/3007018459 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb5ec001c80 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.006+0000 7fb603d8c640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.006+0000 7fb603d8c640 4 set_mon_vals no callback set 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.006+0000 7fb600b26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.006+0000 7fb600b26640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.006+0000 7fb600b26640 10 monclient: dump: 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:48.008 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:48.009 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:48.009 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:48.009 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:48.009 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:48.009 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:48.009 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:48.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.007+0000 7fb603d8c640 10 monclient: shutdown 2024-03-25T18:35:48.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.007+0000 7fb603d8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:48.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.007+0000 7fb603d8c640 1 -- 172.21.15.102:0/3007018459 >> v1:172.21.15.146:6789/0 conn(0x7fb5fc13c9f0 legacy=0x7fb5fc0723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.007+0000 7fb603d8c640 1 -- 172.21.15.102:0/3007018459 shutdown_connections 2024-03-25T18:35:48.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.007+0000 7fb603d8c640 1 -- 172.21.15.102:0/3007018459 >> 172.21.15.102:0/3007018459 conn(0x7fb5fc0b0ef0 msgr2=0x7fb5fc0b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.007+0000 7fb603d8c640 1 -- 172.21.15.102:0/3007018459 shutdown_connections 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.007+0000 7fb603d8c640 1 -- 172.21.15.102:0/3007018459 wait complete. 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.008+0000 7fb603d8c640 10 monclient: build_initial_monmap 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.008+0000 7fb603d8c640 10 monclient: monmap: 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.008+0000 7fb603d8c640 1 librados: starting msgr at 2024-03-25T18:35:48.011 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.008+0000 7fb603d8c640 1 librados: starting objecter 2024-03-25T18:35:48.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.008+0000 7fb603d8c640 1 Processor -- start 2024-03-25T18:35:48.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 1 -- start start 2024-03-25T18:35:48.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 1 librados: setting wanted keys 2024-03-25T18:35:48.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 1 librados: calling monclient init 2024-03-25T18:35:48.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 10 monclient: init 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 10 monclient(hunting): picked mon.noname-b con 0x7fb5fc061830 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 10 monclient(hunting): picked mon.noname-c con 0x7fb5fc072af0 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 10 monclient(hunting): picked mon.noname-a con 0x7fb5fc13c9f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc070210 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc0b4040 con 0x7fb5fc061830 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc1413c0 con 0x7fb5fc072af0 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:48.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.009+0000 7fb603d8c640 10 monclient(hunting): authenticate will time out at 7979.298699s 2024-03-25T18:35:48.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.015+0000 7fb602329640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb5fc13c9f0 0x7fb5fc1e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45454/0 (socket says 172.21.15.102:45454) 2024-03-25T18:35:48.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.015+0000 7fb602329640 1 -- 172.21.15.102:0/579541536 learned_addr learned my addr 172.21.15.102:0/579541536 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:48.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.021+0000 7fb5f27fc640 10 client.?.objecter ms_handle_connect 0x7fb5fc13c9f0 2024-03-25T18:35:48.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.021+0000 7fb5f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:48.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.024+0000 7fb5f27fc640 10 client.?.objecter ms_handle_connect 0x7fb5fc072af0 2024-03-25T18:35:48.026 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.024+0000 7fb5f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:48.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.027+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015774097 0 0) 0x7fb5fc070210 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.028 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.027+0000 7fb5f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:48.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.027+0000 7fb5f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:48.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.027+0000 7fb5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:48.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.027+0000 7fb5f27fc640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:35:48.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.027+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e0004eb0 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.030+0000 7fb5f27fc640 10 client.?.objecter ms_handle_connect 0x7fb5fc061830 2024-03-25T18:35:48.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.030+0000 7fb5f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:48.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.035+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609189106 0 0) 0x7fb5fc1413c0 con 0x7fb5fc072af0 2024-03-25T18:35:48.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.035+0000 7fb5f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:48.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.035+0000 7fb5f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:48.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.035+0000 7fb5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:48.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.036+0000 7fb5f27fc640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:35:48.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.036+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5fc070210 con 0x7fb5fc072af0 2024-03-25T18:35:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.045+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114459263 0 0) 0x7fb5fc0b4040 con 0x7fb5fc061830 2024-03-25T18:35:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.045+0000 7fb5f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:35:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.045+0000 7fb5f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.045+0000 7fb5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.045+0000 7fb5f27fc640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:35:48.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.045+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5fc1413c0 con 0x7fb5fc061830 2024-03-25T18:35:48.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.049+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1149902960 0 0) 0x7fb5e0004eb0 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.050 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.049+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5fc0b4040 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.049+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb5f4001d10 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.049+0000 7fb5f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.049+0000 7fb5f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.049+0000 7fb5f27fc640 10 monclient(hunting): dump: 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:48.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.049+0000 7fb5f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:48.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.051+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1168422418 0 0) 0x7fb5fc070210 con 0x7fb5fc072af0 2024-03-25T18:35:48.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.051+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5e0004eb0 con 0x7fb5fc072af0 2024-03-25T18:35:48.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.051+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb5ec001b80 con 0x7fb5fc072af0 2024-03-25T18:35:48.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.051+0000 7fb5f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:48.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.051+0000 7fb5f27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:35:48.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.051+0000 7fb5f27fc640 10 monclient(hunting): dump: 2024-03-25T18:35:48.052 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:48.053 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:48.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4079944873 0 0) 0x7fb5fc0b4040 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 >> v1:172.21.15.102:6790/0 conn(0x7fb5fc072af0 legacy=0x7fb5fc1d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 >> v1:172.21.15.146:6789/0 conn(0x7fb5fc061830 legacy=0x7fb5fc1d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 1 monclient: found mon.a 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc1402f0 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 10 monclient: _finish_auth 0 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb5f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 5 monclient: authenticate success, global_id 4668 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 10 monclient: _renew_subs 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc1405a0 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 10 client.4668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 10 monclient: _renew_subs 2024-03-25T18:35:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc0b3c90 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.060+0000 7fb603d8c640 1 librados: init done 2024-03-25T18:35:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.061+0000 7fb603d8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.061+0000 7fb603d8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.061+0000 7fb603d8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:48.064 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.061+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc0627b0 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.357 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.355+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5f4003990 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.357 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.355+0000 7fb5f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:48.357 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.355+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fb5f4003ce0 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.357 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.355+0000 7fb5f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:48.357 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.355+0000 7fb5f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:48.357 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.355+0000 7fb5f27fc640 10 monclient: dump: 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:48.358 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:48.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.355+0000 7fb5f3fff640 4 set_mon_vals no callback set 2024-03-25T18:35:48.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.356+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fb5f4003f00 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.357+0000 7fb601b28640 10 monclient: get_auth_request con 0x7fb5e006df70 auth_method 0 2024-03-25T18:35:48.359 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.357+0000 7fb5f27fc640 10 client.4668.objecter ms_handle_connect 0x7fb5e006df70 2024-03-25T18:35:48.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.365+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fb5f4054af0 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.365+0000 7fb5f27fc640 10 client.4668.objecter ms_dispatch 0x7fb5fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:48.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.365+0000 7fb5f27fc640 3 client.4668.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:48.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.365+0000 7fb5f27fc640 3 client.4668.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:48.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.365+0000 7fb5f27fc640 20 client.4668.objecter dump_active .. 0 homeless 2024-03-25T18:35:48.367 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.365+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5f4050600 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.365+0000 7fb5f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:48.368 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.365+0000 7fb5f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:48.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.588+0000 7fb603d8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:48.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.588+0000 7fb603d8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:48.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.588+0000 7fb603d8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:48.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.588+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5fc071050 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.671+0000 7fb5f27fc640 1 -- 172.21.15.102:0/579541536 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fb5f4052530 con 0x7fb5fc13c9f0 2024-03-25T18:35:48.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.671+0000 7fb5f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:48.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.671+0000 7fb5f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:48.674 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:35:48.674 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:48.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 10 librados: watch_flush enter 2024-03-25T18:35:48.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 10 librados: watch_flush exit 2024-03-25T18:35:48.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 20 client.4668.objecter shutdown clearing up homeless session... 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 10 client.4668.objecter successfully canceled tick 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 >> v1:172.21.15.146:6800/3848252714 conn(0x7fb5e006df70 legacy=0x7fb5e0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 10 monclient: shutdown 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 >> v1:172.21.15.102:6789/0 conn(0x7fb5fc13c9f0 legacy=0x7fb5fc1e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 shutdown_connections 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 >> 172.21.15.102:0/579541536 conn(0x7fb5fc0b0ef0 msgr2=0x7fb5fc06e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 shutdown_connections 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 1 -- 172.21.15.102:0/579541536 wait complete. 2024-03-25T18:35:48.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:48.673+0000 7fb603d8c640 1 librados: shutdown 2024-03-25T18:35:48.687 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:35:50.016 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:06.251 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:50.020 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:06.256 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:50.053 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:06.289 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:50.065 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:06.301 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:50.073 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:06.286 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:50.078 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:06.291 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:50.081 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:06.316 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:50.112 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:06.325 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:50.123 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:06.358 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:35:55.494 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:35:55.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.612+0000 7fce24e15640 10 monclient: get_monmap_and_config 2024-03-25T18:35:55.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.612+0000 7fce24e15640 10 monclient: build_initial_monmap 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.612+0000 7fce24e15640 10 monclient: monmap: 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:35:55.612932+0000 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:35:55.612932+0000 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:35:55.614 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:55.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.612+0000 7fce24e15640 1 Processor -- start 2024-03-25T18:35:55.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.613+0000 7fce24e15640 1 -- start start 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.613+0000 7fce24e15640 10 monclient: init 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.613+0000 7fce24e15640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.613+0000 7fce24e15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.613+0000 7fce24e15640 10 monclient(hunting): picked mon.noname-a con 0x7fce200a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.614+0000 7fce24e15640 10 monclient(hunting): picked mon.noname-c con 0x7fce2013f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.614+0000 7fce24e15640 10 monclient(hunting): picked mon.noname-b con 0x7fce201430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.614+0000 7fce24e15640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce20142520 con 0x7fce200a98f0 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.614+0000 7fce24e15640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200b28a0 con 0x7fce2013f710 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.614+0000 7fce24e15640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200b2a30 con 0x7fce201430c0 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.614+0000 7fce24e15640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:55.616 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.614+0000 7fce24e15640 10 monclient(hunting): authenticate will time out at 7986.902228s 2024-03-25T18:35:55.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.624+0000 7fce1ed9d640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fce201430c0 0x7fce2014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:43884/0 (socket says 172.21.15.102:43884) 2024-03-25T18:35:55.624 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.624+0000 7fce1ed9d640 1 -- 172.21.15.102:0/3684819775 learned_addr learned my addr 172.21.15.102:0/3684819775 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.641+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784400129 0 0) 0x7fce200b2a30 con 0x7fce201430c0 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.641+0000 7fce1d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.641+0000 7fce1d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.641+0000 7fce1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.641+0000 7fce1d59a640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:35:55.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.641+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce08003520 con 0x7fce201430c0 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.644+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064175205 0 0) 0x7fce200b28a0 con 0x7fce2013f710 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.644+0000 7fce1d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.644+0000 7fce1d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.644+0000 7fce1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.644+0000 7fce1d59a640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:35:55.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.644+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce200b2a30 con 0x7fce2013f710 2024-03-25T18:35:55.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.649+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995227085 0 0) 0x7fce20142520 con 0x7fce200a98f0 2024-03-25T18:35:55.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.649+0000 7fce1d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.649+0000 7fce1d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.649+0000 7fce1d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.649+0000 7fce1d59a640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:35:55.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.649+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce200b28a0 con 0x7fce200a98f0 2024-03-25T18:35:55.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1128720969 0 0) 0x7fce200b2a30 con 0x7fce2013f710 2024-03-25T18:35:55.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 >> v1:172.21.15.102:6790/0 conn(0x7fce201430c0 legacy=0x7fce2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 >> v1:172.21.15.102:6789/0 conn(0x7fce200a98f0 legacy=0x7fce200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:35:55.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 1 monclient: found mon.noname-c 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce200b44c0 con 0x7fce2013f710 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 10 monclient: _finish_auth 0 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fce14001670 con 0x7fce2013f710 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce1d59a640 10 monclient: dump: 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:55.654 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:55.655 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:55.655 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:55.655 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:55.655 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:55.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce24e15640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:35:55.655 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.652+0000 7fce24e15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:35:55.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.657+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce140017d0 con 0x7fce2013f710 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.657+0000 7fce1d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.657+0000 7fce1d59a640 1 -- 172.21.15.102:0/3684819775 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fce14001be0 con 0x7fce2013f710 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.657+0000 7fce1d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.657+0000 7fce1d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.657+0000 7fce1d59a640 10 monclient: dump: 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:55.658 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:55.659 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:55.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.657+0000 7fce24e15640 10 monclient: get_monmap_and_config success 2024-03-25T18:35:55.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.657+0000 7fce24e15640 4 set_mon_vals no callback set 2024-03-25T18:35:55.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.658+0000 7fce24e15640 10 monclient: shutdown 2024-03-25T18:35:55.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.658+0000 7fce24e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:55.659 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.658+0000 7fce24e15640 1 -- 172.21.15.102:0/3684819775 >> v1:172.21.15.146:6789/0 conn(0x7fce2013f710 legacy=0x7fce201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.658+0000 7fce24e15640 1 -- 172.21.15.102:0/3684819775 shutdown_connections 2024-03-25T18:35:55.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.658+0000 7fce24e15640 1 -- 172.21.15.102:0/3684819775 >> 172.21.15.102:0/3684819775 conn(0x7fce200b7ea0 msgr2=0x7fce200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:55.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.658+0000 7fce24e15640 1 -- 172.21.15.102:0/3684819775 shutdown_connections 2024-03-25T18:35:55.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.658+0000 7fce24e15640 1 -- 172.21.15.102:0/3684819775 wait complete. 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.659+0000 7fce24e15640 10 monclient: build_initial_monmap 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.659+0000 7fce24e15640 10 monclient: monmap: 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.660+0000 7fce24e15640 1 librados: starting msgr at 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.660+0000 7fce24e15640 1 librados: starting objecter 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.660+0000 7fce24e15640 1 Processor -- start 2024-03-25T18:35:55.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.660+0000 7fce24e15640 1 -- start start 2024-03-25T18:35:55.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.660+0000 7fce24e15640 1 librados: setting wanted keys 2024-03-25T18:35:55.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.660+0000 7fce24e15640 1 librados: calling monclient init 2024-03-25T18:35:55.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.660+0000 7fce24e15640 10 monclient: init 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 10 monclient: _reopen_session rank -1 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 10 monclient(hunting): picked mon.noname-c con 0x7fce200a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 10 monclient(hunting): picked mon.noname-b con 0x7fce2013f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 10 monclient(hunting): picked mon.noname-a con 0x7fce201430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce201d7a90 con 0x7fce201430c0 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200b1700 con 0x7fce2013f710 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce200b13d0 con 0x7fce200a98f0 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 10 monclient(hunting): _renew_subs 2024-03-25T18:35:55.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.661+0000 7fce24e15640 10 monclient(hunting): authenticate will time out at 7986.949308s 2024-03-25T18:35:55.666 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.666+0000 7fce1ed9d640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fce201430c0 0x7fce2014d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35108/0 (socket says 172.21.15.102:35108) 2024-03-25T18:35:55.667 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.666+0000 7fce1ed9d640 1 -- 172.21.15.102:0/3738834174 learned_addr learned my addr 172.21.15.102:0/3738834174 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:35:55.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.668+0000 7fce0effd640 10 client.?.objecter ms_handle_connect 0x7fce201430c0 2024-03-25T18:35:55.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.668+0000 7fce0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:55.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.676+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195201847 0 0) 0x7fce201d7a90 con 0x7fce201430c0 2024-03-25T18:35:55.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.676+0000 7fce0effd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.676+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.676+0000 7fce0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.676+0000 7fce0effd640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:35:55.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.677+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce00004eb0 con 0x7fce201430c0 2024-03-25T18:35:55.679 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.678+0000 7fce0effd640 10 client.?.objecter ms_handle_connect 0x7fce2013f710 2024-03-25T18:35:55.679 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.678+0000 7fce0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:55.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.688+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955471694 0 0) 0x7fce200b1700 con 0x7fce2013f710 2024-03-25T18:35:55.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.688+0000 7fce0effd640 10 monclient(hunting): _negotiate 2024-03-25T18:35:55.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.688+0000 7fce0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:35:55.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.688+0000 7fce0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:35:55.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.688+0000 7fce0effd640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:35:55.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.688+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce201d7a90 con 0x7fce2013f710 2024-03-25T18:35:55.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.692+0000 7fce0effd640 10 client.?.objecter ms_handle_connect 0x7fce200a98f0 2024-03-25T18:35:55.693 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.692+0000 7fce0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:35:55.702 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.701+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3696182255 0 0) 0x7fce00004eb0 con 0x7fce201430c0 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.701+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce200b1700 con 0x7fce201430c0 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.701+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fce18001b30 con 0x7fce201430c0 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.701+0000 7fce0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.701+0000 7fce0effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.701+0000 7fce0effd640 10 monclient(hunting): dump: 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:55.703 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:55.704 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:55.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.701+0000 7fce0effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:35:55.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.717+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3632388711 0 0) 0x7fce201d7a90 con 0x7fce2013f710 2024-03-25T18:35:55.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.717+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce00004eb0 con 0x7fce2013f710 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.717+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fce14001b20 con 0x7fce2013f710 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.717+0000 7fce0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.717+0000 7fce0effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.717+0000 7fce0effd640 10 monclient(hunting): dump: 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:55.718 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2407722616 0 0) 0x7fce200b1700 con 0x7fce201430c0 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 >> v1:172.21.15.102:6790/0 conn(0x7fce200a98f0 legacy=0x7fce201d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 >> v1:172.21.15.146:6789/0 conn(0x7fce2013f710 legacy=0x7fce201d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 10 monclient: _finish_hunting 0 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 1 monclient: found mon.a 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2014eb50 con 0x7fce201430c0 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 10 monclient: _finish_auth 0 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 10 monclient: _check_auth_tickets 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 5 monclient: authenticate success, global_id 4674 2024-03-25T18:35:55.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 10 monclient: _renew_subs 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2014dc50 con 0x7fce201430c0 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 10 client.4674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 10 monclient: _renew_subs 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2014e290 con 0x7fce201430c0 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.721+0000 7fce24e15640 1 librados: init done 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.722+0000 7fce24e15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.722+0000 7fce24e15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:55.723 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.722+0000 7fce24e15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:55.724 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:55.722+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce200b1560 con 0x7fce201430c0 2024-03-25T18:35:56.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.032+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce18001fe0 con 0x7fce201430c0 2024-03-25T18:35:56.033 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.033+0000 7fce0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.033+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fce18004b60 con 0x7fce201430c0 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.033+0000 7fce0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.033+0000 7fce0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.033+0000 7fce0effd640 10 monclient: dump: 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:35:56.034 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.033+0000 7fce1cd99640 4 set_mon_vals no callback set 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.033+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fce1801ab70 con 0x7fce201430c0 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.034+0000 7fce1e59c640 10 monclient: get_auth_request con 0x7fce0006de40 auth_method 0 2024-03-25T18:35:56.035 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.034+0000 7fce0effd640 10 client.4674.objecter ms_handle_connect 0x7fce0006de40 2024-03-25T18:35:56.040 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.040+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7fce180548e0 con 0x7fce201430c0 2024-03-25T18:35:56.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.040+0000 7fce0effd640 10 client.4674.objecter ms_dispatch 0x7fce20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:35:56.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.040+0000 7fce0effd640 3 client.4674.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:35:56.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.040+0000 7fce0effd640 3 client.4674.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:35:56.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.040+0000 7fce0effd640 20 client.4674.objecter dump_active .. 0 homeless 2024-03-25T18:35:56.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.040+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce18050220 con 0x7fce201430c0 2024-03-25T18:35:56.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.040+0000 7fce0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:35:56.041 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.040+0000 7fce0effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:35:56.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.289+0000 7fce24e15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:56.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.289+0000 7fce24e15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:56.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.289+0000 7fce24e15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:35:56.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.289+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce200b0d00 con 0x7fce201430c0 2024-03-25T18:35:56.327 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.326+0000 7fce0effd640 1 -- 172.21.15.102:0/3738834174 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7fce18052320 con 0x7fce201430c0 2024-03-25T18:35:56.327 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.326+0000 7fce0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:35:56.327 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.326+0000 7fce0effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:35:56.327 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:35:56.327 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.328+0000 7fce24e15640 10 librados: watch_flush enter 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.328+0000 7fce24e15640 10 librados: watch_flush exit 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.328+0000 7fce24e15640 20 client.4674.objecter shutdown clearing up homeless session... 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.328+0000 7fce24e15640 10 client.4674.objecter successfully canceled tick 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.328+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 >> v1:172.21.15.146:6800/3848252714 conn(0x7fce0006de40 legacy=0x7fce00070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.328+0000 7fce24e15640 10 monclient: shutdown 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.328+0000 7fce24e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.328+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 >> v1:172.21.15.102:6789/0 conn(0x7fce201430c0 legacy=0x7fce2014d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.329+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 shutdown_connections 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.329+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 >> 172.21.15.102:0/3738834174 conn(0x7fce200b7ea0 msgr2=0x7fce20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.329+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 shutdown_connections 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.329+0000 7fce24e15640 1 -- 172.21.15.102:0/3738834174 wait complete. 2024-03-25T18:35:56.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:35:56.329+0000 7fce24e15640 1 librados: shutdown 2024-03-25T18:35:56.342 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:03.148 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:03.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.263+0000 7f7837bcf640 10 monclient: get_monmap_and_config 2024-03-25T18:36:03.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.263+0000 7f7837bcf640 10 monclient: build_initial_monmap 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.263+0000 7f7837bcf640 10 monclient: monmap: 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:36:03.263742+0000 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:36:03.263742+0000 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:36:03.264 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:03.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.263+0000 7f7837bcf640 1 Processor -- start 2024-03-25T18:36:03.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.263+0000 7f7837bcf640 1 -- start start 2024-03-25T18:36:03.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.263+0000 7f7837bcf640 10 monclient: init 2024-03-25T18:36:03.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f78300b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f78300b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f783014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78300af940 con 0x7f78300b4d60 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78300b0a70 con 0x7f78300b1730 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78300b0d50 con 0x7f783014be90 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:03.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.264+0000 7f7837bcf640 10 monclient(hunting): authenticate will time out at 7994.552633s 2024-03-25T18:36:03.268 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.267+0000 7f783596b640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f78300b4d60 0x7f783013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43782/0 (socket says 172.21.15.102:43782) 2024-03-25T18:36:03.268 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.267+0000 7f783596b640 1 -- 172.21.15.102:0/2998486111 learned_addr learned my addr 172.21.15.102:0/2998486111 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:03.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.282+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153459818 0 0) 0x7f78300af940 con 0x7f78300b4d60 2024-03-25T18:36:03.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.282+0000 7f7834969640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.282+0000 7f7834969640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.282+0000 7f7834969640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.282+0000 7f7834969640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:36:03.283 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.282+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f781c003520 con 0x7f78300b4d60 2024-03-25T18:36:03.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.284+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650392546 0 0) 0x7f78300b0d50 con 0x7f783014be90 2024-03-25T18:36:03.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.284+0000 7f7834969640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.284+0000 7f7834969640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.284+0000 7f7834969640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.284+0000 7f7834969640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:36:03.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.284+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78300af940 con 0x7f783014be90 2024-03-25T18:36:03.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4004523974 0 0) 0x7f78300af940 con 0x7f783014be90 2024-03-25T18:36:03.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 >> v1:172.21.15.146:6789/0 conn(0x7f78300b1730 legacy=0x7f78300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 >> v1:172.21.15.102:6789/0 conn(0x7f78300b4d60 legacy=0x7f783013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 10 monclient: _finish_hunting 0 2024-03-25T18:36:03.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 1 monclient: found mon.noname-b 2024-03-25T18:36:03.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78300b1030 con 0x7f783014be90 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 10 monclient: _finish_auth 0 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 10 monclient: _check_auth_tickets 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f78280019f0 con 0x7f783014be90 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7834969640 10 monclient: dump: 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.300 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:03.301 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:03.301 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:03.301 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:03.301 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:03.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7837bcf640 5 monclient: authenticate success, global_id 4661 2024-03-25T18:36:03.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.298+0000 7f7837bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:03.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.312+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7828001b50 con 0x7f783014be90 2024-03-25T18:36:03.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.312+0000 7f7834969640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:03.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.312+0000 7f7834969640 1 -- 172.21.15.102:0/2998486111 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7828001f20 con 0x7f783014be90 2024-03-25T18:36:03.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.312+0000 7f7837bcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:03.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.312+0000 7f7837bcf640 4 set_mon_vals no callback set 2024-03-25T18:36:03.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.312+0000 7f7834969640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.312+0000 7f7834969640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.312+0000 7f7834969640 10 monclient: dump: 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:03.314 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.313+0000 7f7837bcf640 10 monclient: shutdown 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.313+0000 7f7837bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.313+0000 7f7837bcf640 1 -- 172.21.15.102:0/2998486111 >> v1:172.21.15.102:6790/0 conn(0x7f783014be90 legacy=0x7f783014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.313+0000 7f7837bcf640 1 -- 172.21.15.102:0/2998486111 shutdown_connections 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.313+0000 7f7837bcf640 1 -- 172.21.15.102:0/2998486111 >> 172.21.15.102:0/2998486111 conn(0x7f78300617a0 msgr2=0x7f7830061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.313+0000 7f7837bcf640 1 -- 172.21.15.102:0/2998486111 shutdown_connections 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.313+0000 7f7837bcf640 1 -- 172.21.15.102:0/2998486111 wait complete. 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.314+0000 7f7837bcf640 10 monclient: build_initial_monmap 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.314+0000 7f7837bcf640 10 monclient: monmap: 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:36:03.316 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 1 librados: starting msgr at 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 1 librados: starting objecter 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 1 Processor -- start 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 1 -- start start 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 1 librados: setting wanted keys 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 1 librados: calling monclient init 2024-03-25T18:36:03.317 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 10 monclient: init 2024-03-25T18:36:03.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:03.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.315+0000 7f7837bcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:03.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.316+0000 7f7837bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f78300b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:03.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.316+0000 7f7837bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f78300b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:03.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.316+0000 7f7837bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f783014be90 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:03.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.316+0000 7f7837bcf640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7830138340 con 0x7f78300b4d60 2024-03-25T18:36:03.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.316+0000 7f7837bcf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7830137a50 con 0x7f78300b1730 2024-03-25T18:36:03.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.316+0000 7f7837bcf640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7830137be0 con 0x7f783014be90 2024-03-25T18:36:03.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.316+0000 7f7837bcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:03.319 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.316+0000 7f7837bcf640 10 monclient(hunting): authenticate will time out at 7994.604159s 2024-03-25T18:36:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.322+0000 7f783596b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f78300b1730 0x7f7830131d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:58138/0 (socket says 172.21.15.102:58138) 2024-03-25T18:36:03.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.322+0000 7f783596b640 1 -- 172.21.15.102:0/3532954288 learned_addr learned my addr 172.21.15.102:0/3532954288 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:03.326 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.326+0000 7f78267fc640 10 client.?.objecter ms_handle_connect 0x7f78300b1730 2024-03-25T18:36:03.326 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.326+0000 7f78267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:03.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.333+0000 7f78267fc640 10 client.?.objecter ms_handle_connect 0x7f78300b4d60 2024-03-25T18:36:03.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.333+0000 7f78267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:03.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.337+0000 7f78267fc640 10 client.?.objecter ms_handle_connect 0x7f783014be90 2024-03-25T18:36:03.338 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.337+0000 7f78267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:03.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.380+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763884426 0 0) 0x7f7830137be0 con 0x7f783014be90 2024-03-25T18:36:03.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.380+0000 7f78267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.380+0000 7f78267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.380+0000 7f78267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.380+0000 7f78267fc640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:36:03.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.380+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7814004eb0 con 0x7f783014be90 2024-03-25T18:36:03.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.453+0000 7f783616c640 1 -- 172.21.15.102:0/3532954288 >> v1:172.21.15.102:6790/0 conn(0x7f783014be90 legacy=0x7f78301e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:36:03.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.453+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3191894867 0 0) 0x7f7814004eb0 con 0x7f783014be90 2024-03-25T18:36:03.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.453+0000 7f783616c640 1 -- 172.21.15.102:0/3532954288 >> v1:172.21.15.102:6790/0 conn(0x7f783014be90 legacy=0x7f78301e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:36:03.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.453+0000 7f783616c640 1 --1- 172.21.15.102:0/3532954288 >> v1:172.21.15.102:6790/0 conn(0x7f783014be90 0x7f78301e2ed0 :-1 s=OPENED pgs=75271 cs=1 l=1).handle_message read tag failed 2024-03-25T18:36:03.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.453+0000 7f783616c640 1 --1- 172.21.15.102:0/3532954288 >> v1:172.21.15.102:6790/0 conn(0x7f783014be90 0x7f78301e2ed0 :-1 s=OPENED pgs=75271 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:36:03.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.453+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7830137be0 con 0x7f783014be90 2024-03-25T18:36:03.454 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.453+0000 7f78267fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.102:6790/0 2024-03-25T18:36:03.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.487+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171060055 0 0) 0x7f7830138340 con 0x7f78300b4d60 2024-03-25T18:36:03.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.487+0000 7f78267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.487+0000 7f78267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.487+0000 7f78267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.487+0000 7f78267fc640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:36:03.488 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.487+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7814004eb0 con 0x7f78300b4d60 2024-03-25T18:36:03.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.496+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987287931 0 0) 0x7f7830137a50 con 0x7f78300b1730 2024-03-25T18:36:03.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.496+0000 7f78267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:36:03.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.496+0000 7f78267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:03.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.496+0000 7f78267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:03.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.496+0000 7f78267fc640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:36:03.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.496+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7830138340 con 0x7f78300b1730 2024-03-25T18:36:03.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.511+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3237391415 0 0) 0x7f7814004eb0 con 0x7f78300b4d60 2024-03-25T18:36:03.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.511+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7830137a50 con 0x7f78300b4d60 2024-03-25T18:36:03.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.511+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7820001870 con 0x7f78300b4d60 2024-03-25T18:36:03.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.511+0000 7f78267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.511+0000 7f78267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:03.512 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.511+0000 7f78267fc640 10 monclient(hunting): dump: 2024-03-25T18:36:03.512 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:03.513 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.511+0000 7f78267fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:03.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.527+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1905413614 0 0) 0x7f7830138340 con 0x7f78300b1730 2024-03-25T18:36:03.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.527+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7814004eb0 con 0x7f78300b1730 2024-03-25T18:36:03.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.527+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f782c001950 con 0x7f78300b1730 2024-03-25T18:36:03.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.527+0000 7f78267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.527+0000 7f78267fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:03.528 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.527+0000 7f78267fc640 10 monclient(hunting): dump: 2024-03-25T18:36:03.528 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:03.529 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1689105948 0 0) 0x7f7830137a50 con 0x7f78300b4d60 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 >> v1:172.21.15.102:6790/0 conn(0x7f783014be90 legacy=0x7f78301e2ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 >> v1:172.21.15.146:6789/0 conn(0x7f78300b1730 legacy=0x7f7830131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 1 monclient: found mon.a 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830140380 con 0x7f78300b4d60 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 10 monclient: _finish_auth 0 2024-03-25T18:36:03.535 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f78267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 5 monclient: authenticate success, global_id 4680 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 10 monclient: _renew_subs 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78301405d0 con 0x7f78300b4d60 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 10 client.4680.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 10 monclient: _renew_subs 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7830144f20 con 0x7f78300b4d60 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.533+0000 7f7837bcf640 1 librados: init done 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.534+0000 7f7837bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.534+0000 7f7837bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.534+0000 7f7837bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:03.536 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.534+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78300627b0 con 0x7f78300b4d60 2024-03-25T18:36:03.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.852+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78200035f0 con 0x7f78300b4d60 2024-03-25T18:36:03.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.852+0000 7f78267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:03.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.852+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f7820001d80 con 0x7f78300b4d60 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.852+0000 7f78267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.852+0000 7f78267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.852+0000 7f78267fc640 10 monclient: dump: 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:03.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.852+0000 7f7827fff640 4 set_mon_vals no callback set 2024-03-25T18:36:03.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.853+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f78200197f0 con 0x7f78300b4d60 2024-03-25T18:36:03.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.854+0000 7f783596b640 10 monclient: get_auth_request con 0x7f781406e120 auth_method 0 2024-03-25T18:36:03.855 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.855+0000 7f78267fc640 10 client.4680.objecter ms_handle_connect 0x7f781406e120 2024-03-25T18:36:03.860 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.860+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 440042915 0 0) 0x7f7820053500 con 0x7f78300b4d60 2024-03-25T18:36:03.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.860+0000 7f78267fc640 10 client.4680.objecter ms_dispatch 0x7f7830000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:36:03.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.860+0000 7f78267fc640 3 client.4680.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:36:03.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.860+0000 7f78267fc640 3 client.4680.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:36:03.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.860+0000 7f78267fc640 20 client.4680.objecter dump_active .. 0 homeless 2024-03-25T18:36:03.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.860+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f782004ee40 con 0x7f78300b4d60 2024-03-25T18:36:03.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.860+0000 7f78267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:03.861 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:03.860+0000 7f78267fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:04.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.125+0000 7f7837bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:04.125 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.125+0000 7f7837bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:04.126 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.125+0000 7f7837bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:04.126 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.125+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7830069f40 con 0x7f78300b4d60 2024-03-25T18:36:04.166 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.166+0000 7f78267fc640 1 -- 172.21.15.102:0/3532954288 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2459799212) 0x7f7820050f40 con 0x7f78300b4d60 2024-03-25T18:36:04.166 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.166+0000 7f78267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:04.167 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.166+0000 7f78267fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:04.167 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:36:04.167 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:26:06.078026+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:06.078026+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:04.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 10 librados: watch_flush enter 2024-03-25T18:36:04.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 10 librados: watch_flush exit 2024-03-25T18:36:04.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 20 client.4680.objecter shutdown clearing up homeless session... 2024-03-25T18:36:04.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 10 client.4680.objecter successfully canceled tick 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 >> v1:172.21.15.146:6800/3848252714 conn(0x7f781406e120 legacy=0x7f78140705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 10 monclient: shutdown 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 >> v1:172.21.15.102:6789/0 conn(0x7f78300b4d60 legacy=0x7f78301dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 shutdown_connections 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 >> 172.21.15.102:0/3532954288 conn(0x7f78300617a0 msgr2=0x7f78300b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 shutdown_connections 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 1 -- 172.21.15.102:0/3532954288 wait complete. 2024-03-25T18:36:04.170 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:04.168+0000 7f7837bcf640 1 librados: shutdown 2024-03-25T18:36:04.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:10.046 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:26.282 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.096 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:26.331 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.124 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:26.359 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.161 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:26.374 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.165 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:26.379 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.167 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:26.403 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.176 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:26.412 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.196 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:26.409 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.207 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:26.443 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:10.989 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:11.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.104+0000 7f4ca56ed640 10 monclient: get_monmap_and_config 2024-03-25T18:36:11.104 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.104+0000 7f4ca56ed640 10 monclient: build_initial_monmap 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.104+0000 7f4ca56ed640 10 monclient: monmap: 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:36:11.104813+0000 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:36:11.104813+0000 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:36:11.105 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:11.106 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.104+0000 7f4ca56ed640 1 Processor -- start 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.104+0000 7f4ca56ed640 1 -- start start 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.104+0000 7f4ca56ed640 10 monclient: init 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 10 monclient(hunting): picked mon.noname-b con 0x7f4ca00a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 10 monclient(hunting): picked mon.noname-c con 0x7f4ca013f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca01430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca0142520 con 0x7f4ca01430c0 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca00b28a0 con 0x7f4ca013f710 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca00b2a30 con 0x7f4ca00a98f0 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:11.107 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.105+0000 7f4ca56ed640 10 monclient(hunting): authenticate will time out at 8002.393645s 2024-03-25T18:36:11.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.116+0000 7f4c9effd640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4ca00a98f0 0x7f4ca00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:42864/0 (socket says 172.21.15.102:42864) 2024-03-25T18:36:11.117 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.116+0000 7f4c9effd640 1 -- 172.21.15.102:0/4226918602 learned_addr learned my addr 172.21.15.102:0/4226918602 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:11.128 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.127+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6968981 0 0) 0x7f4ca00b2a30 con 0x7f4ca00a98f0 2024-03-25T18:36:11.128 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.127+0000 7f4c9dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.128 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.127+0000 7f4c9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.128 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.127+0000 7f4c9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.128 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.127+0000 7f4c9dffb640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:36:11.128 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.127+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c88003520 con 0x7f4ca00a98f0 2024-03-25T18:36:11.134 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.134+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444650865 0 0) 0x7f4ca0142520 con 0x7f4ca01430c0 2024-03-25T18:36:11.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.134+0000 7f4c9dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.134+0000 7f4c9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.134+0000 7f4c9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.134+0000 7f4c9dffb640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:36:11.135 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.134+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca00b2a30 con 0x7f4ca01430c0 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4244669569 0 0) 0x7f4c88003520 con 0x7f4ca00a98f0 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 >> v1:172.21.15.146:6789/0 conn(0x7f4ca013f710 legacy=0x7f4ca01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 >> v1:172.21.15.102:6789/0 conn(0x7f4ca01430c0 legacy=0x7f4ca014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 1 monclient: found mon.noname-b 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:36:11.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca00b44c0 con 0x7f4ca00a98f0 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 10 monclient: _finish_auth 0 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4c90001690 con 0x7f4ca00a98f0 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4c9dffb640 10 monclient: dump: 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:11.140 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:11.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4ca56ed640 5 monclient: authenticate success, global_id 4667 2024-03-25T18:36:11.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.138+0000 7f4ca56ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:11.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.146+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c900017f0 con 0x7f4ca00a98f0 2024-03-25T18:36:11.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.146+0000 7f4c9dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:11.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.146+0000 7f4c9dffb640 1 -- 172.21.15.102:0/4226918602 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4c90001c00 con 0x7f4ca00a98f0 2024-03-25T18:36:11.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.146+0000 7f4ca56ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:11.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.146+0000 7f4ca56ed640 4 set_mon_vals no callback set 2024-03-25T18:36:11.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.146+0000 7f4c9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:11.147 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.146+0000 7f4c9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.146+0000 7f4c9dffb640 10 monclient: dump: 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:11.148 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:11.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.147+0000 7f4ca56ed640 10 monclient: shutdown 2024-03-25T18:36:11.149 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.147+0000 7f4ca56ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.147+0000 7f4ca56ed640 1 -- 172.21.15.102:0/4226918602 >> v1:172.21.15.102:6790/0 conn(0x7f4ca00a98f0 legacy=0x7f4ca00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.147+0000 7f4ca56ed640 1 -- 172.21.15.102:0/4226918602 shutdown_connections 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.147+0000 7f4ca56ed640 1 -- 172.21.15.102:0/4226918602 >> 172.21.15.102:0/4226918602 conn(0x7f4ca00b7ea0 msgr2=0x7f4ca00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.147+0000 7f4ca56ed640 1 -- 172.21.15.102:0/4226918602 shutdown_connections 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.147+0000 7f4ca56ed640 1 -- 172.21.15.102:0/4226918602 wait complete. 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.148+0000 7f4ca56ed640 10 monclient: build_initial_monmap 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.148+0000 7f4ca56ed640 10 monclient: monmap: 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:36:11.150 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.148+0000 7f4ca56ed640 1 librados: starting msgr at 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.148+0000 7f4ca56ed640 1 librados: starting objecter 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 1 Processor -- start 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 1 -- start start 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 1 librados: setting wanted keys 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 1 librados: calling monclient init 2024-03-25T18:36:11.151 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 10 monclient: init 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 10 monclient(hunting): picked mon.noname-a con 0x7f4ca00a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 10 monclient(hunting): picked mon.noname-b con 0x7f4ca013f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 10 monclient(hunting): picked mon.noname-c con 0x7f4ca01430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca013be70 con 0x7f4ca00a98f0 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca0131660 con 0x7f4ca013f710 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.149+0000 7f4ca56ed640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca01317f0 con 0x7f4ca01430c0 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.150+0000 7f4ca56ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:11.152 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.150+0000 7f4ca56ed640 10 monclient(hunting): authenticate will time out at 8002.437954s 2024-03-25T18:36:11.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.160+0000 7f4c9e7fc640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4ca013f710 0x7f4ca01d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:34750/0 (socket says 172.21.15.102:34750) 2024-03-25T18:36:11.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.160+0000 7f4c9e7fc640 1 -- 172.21.15.102:0/3913380042 learned_addr learned my addr 172.21.15.102:0/3913380042 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:11.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.162+0000 7f4c877fe640 10 client.?.objecter ms_handle_connect 0x7f4ca013f710 2024-03-25T18:36:11.162 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.162+0000 7f4c877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:11.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.173+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032443030 0 0) 0x7f4ca0131660 con 0x7f4ca013f710 2024-03-25T18:36:11.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.173+0000 7f4c877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.173+0000 7f4c877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.173+0000 7f4c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.173+0000 7f4c877fe640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:36:11.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.173+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c004eb0 con 0x7f4ca013f710 2024-03-25T18:36:11.183 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.182+0000 7f4c877fe640 10 client.?.objecter ms_handle_connect 0x7f4ca00a98f0 2024-03-25T18:36:11.183 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.182+0000 7f4c877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:11.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.183+0000 7f4c877fe640 10 client.?.objecter ms_handle_connect 0x7f4ca01430c0 2024-03-25T18:36:11.184 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.183+0000 7f4c877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:11.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.189+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226479222 0 0) 0x7f4c8c004eb0 con 0x7f4ca013f710 2024-03-25T18:36:11.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.189+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ca0131660 con 0x7f4ca013f710 2024-03-25T18:36:11.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.189+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4c940019d0 con 0x7f4ca013f710 2024-03-25T18:36:11.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.189+0000 7f4c877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:11.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.189+0000 7f4c877fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:11.190 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.189+0000 7f4c877fe640 10 monclient(hunting): dump: 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:11.191 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.189+0000 7f4c877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:11.192 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.192+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529530284 0 0) 0x7f4ca013be70 con 0x7f4ca00a98f0 2024-03-25T18:36:11.193 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.192+0000 7f4c877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.193 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.192+0000 7f4c877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.193 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.192+0000 7f4c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.193 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.192+0000 7f4c877fe640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:36:11.193 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.192+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c004eb0 con 0x7f4ca00a98f0 2024-03-25T18:36:11.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.193+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883851090 0 0) 0x7f4ca01317f0 con 0x7f4ca01430c0 2024-03-25T18:36:11.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.193+0000 7f4c877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:11.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.193+0000 7f4c877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:11.194 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.193+0000 7f4c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:11.195 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.193+0000 7f4c877fe640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:36:11.195 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.193+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca013be70 con 0x7f4ca01430c0 2024-03-25T18:36:11.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3473935239 0 0) 0x7f4ca0131660 con 0x7f4ca013f710 2024-03-25T18:36:11.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 >> v1:172.21.15.102:6790/0 conn(0x7f4ca01430c0 legacy=0x7f4ca01e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 >> v1:172.21.15.102:6789/0 conn(0x7f4ca00a98f0 legacy=0x7f4ca0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.197 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 1 monclient: found mon.b 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0131980 con 0x7f4ca013f710 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4c877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 10 monclient: _renew_subs 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca0135c60 con 0x7f4ca013f710 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 10 monclient: _renew_subs 2024-03-25T18:36:11.198 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:11.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca00b7770 con 0x7f4ca013f710 2024-03-25T18:36:11.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.196+0000 7f4ca56ed640 1 librados: init done 2024-03-25T18:36:11.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.197+0000 7f4ca56ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:11.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.197+0000 7f4ca56ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:11.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.197+0000 7f4ca56ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:11.199 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.197+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca01371d0 con 0x7f4ca013f710 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.467+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c94001c50 con 0x7f4ca013f710 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.467+0000 7f4c877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.467+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4c94003a60 con 0x7f4ca013f710 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.467+0000 7f4c877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.467+0000 7f4c877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.467+0000 7f4c877fe640 10 monclient: dump: 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:11.468 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:11.469 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:11.469 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:11.469 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:11.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.467+0000 7f4c9d7fa640 4 set_mon_vals no callback set 2024-03-25T18:36:11.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.467+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f4c94003c80 con 0x7f4ca013f710 2024-03-25T18:36:11.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.468+0000 7f4c9effd640 10 monclient: get_auth_request con 0x7f4c8c06dea0 auth_method 0 2024-03-25T18:36:11.469 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.468+0000 7f4c877fe640 10 client.4693.objecter ms_handle_connect 0x7f4c8c06dea0 2024-03-25T18:36:11.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.474+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7f4c94054830 con 0x7f4ca013f710 2024-03-25T18:36:11.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.474+0000 7f4c877fe640 10 client.4693.objecter ms_dispatch 0x7f4ca0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:11.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.474+0000 7f4c877fe640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:11.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.474+0000 7f4c877fe640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:11.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.474+0000 7f4c877fe640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T18:36:11.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.474+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c94050340 con 0x7f4ca013f710 2024-03-25T18:36:11.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.474+0000 7f4c877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:11.475 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.474+0000 7f4c877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:11.709 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.709+0000 7f4ca56ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:11.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.709+0000 7f4ca56ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:11.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.709+0000 7f4ca56ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:11.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.709+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ca00b7ac0 con 0x7f4ca013f710 2024-03-25T18:36:11.731 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.730+0000 7f4c877fe640 1 -- 172.21.15.102:0/3913380042 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7f4c94052270 con 0x7f4ca013f710 2024-03-25T18:36:11.731 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.730+0000 7f4c877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:11.731 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.730+0000 7f4c877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:11.731 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:36:11.732 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:11.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 10 librados: watch_flush enter 2024-03-25T18:36:11.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 10 librados: watch_flush exit 2024-03-25T18:36:11.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T18:36:11.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 10 client.4693.objecter successfully canceled tick 2024-03-25T18:36:11.734 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 >> v1:172.21.15.146:6800/3848252714 conn(0x7f4c8c06dea0 legacy=0x7f4c8c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 10 monclient: shutdown 2024-03-25T18:36:11.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:11.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 >> v1:172.21.15.146:6789/0 conn(0x7f4ca013f710 legacy=0x7f4ca01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:11.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 shutdown_connections 2024-03-25T18:36:11.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 >> 172.21.15.102:0/3913380042 conn(0x7f4ca00b7ea0 msgr2=0x7f4ca0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:11.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 shutdown_connections 2024-03-25T18:36:11.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 1 -- 172.21.15.102:0/3913380042 wait complete. 2024-03-25T18:36:11.735 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:11.733+0000 7f4ca56ed640 1 librados: shutdown 2024-03-25T18:36:11.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:18.551 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:18.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.659+0000 7fc82a2e6640 10 monclient: get_monmap_and_config 2024-03-25T18:36:18.660 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.659+0000 7fc82a2e6640 10 monclient: build_initial_monmap 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.659+0000 7fc82a2e6640 10 monclient: monmap: 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:36:18.660492+0000 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:36:18.660492+0000 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:36:18.661 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:18.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.660+0000 7fc82a2e6640 1 Processor -- start 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.660+0000 7fc82a2e6640 1 -- start start 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.660+0000 7fc82a2e6640 10 monclient: init 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.661+0000 7fc82a2e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.661+0000 7fc82a2e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.661+0000 7fc82a2e6640 10 monclient(hunting): picked mon.noname-b con 0x7fc8240a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.661+0000 7fc82a2e6640 10 monclient(hunting): picked mon.noname-c con 0x7fc82413f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.661+0000 7fc82a2e6640 10 monclient(hunting): picked mon.noname-a con 0x7fc8241430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.661+0000 7fc82a2e6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc824142520 con 0x7fc8241430c0 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.661+0000 7fc82a2e6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8240b28a0 con 0x7fc82413f710 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.662+0000 7fc82a2e6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8240b2a30 con 0x7fc8240a98f0 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.662+0000 7fc82a2e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:18.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.662+0000 7fc82a2e6640 10 monclient(hunting): authenticate will time out at 8009.950158s 2024-03-25T18:36:18.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.670+0000 7fc828883640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc8241430c0 0x7fc82414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39892/0 (socket says 172.21.15.102:39892) 2024-03-25T18:36:18.671 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.671+0000 7fc828883640 1 -- 172.21.15.102:0/2876779981 learned_addr learned my addr 172.21.15.102:0/2876779981 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:18.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.686+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920388534 0 0) 0x7fc824142520 con 0x7fc8241430c0 2024-03-25T18:36:18.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.686+0000 7fc822ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:18.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.686+0000 7fc822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:18.686 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.686+0000 7fc822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:18.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.686+0000 7fc822ffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:36:18.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.686+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc80c003520 con 0x7fc8241430c0 2024-03-25T18:36:18.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.703+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648667723 0 0) 0x7fc8240b28a0 con 0x7fc82413f710 2024-03-25T18:36:18.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.703+0000 7fc822ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:18.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.703+0000 7fc822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:18.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.703+0000 7fc822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:18.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.703+0000 7fc822ffd640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:36:18.704 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.703+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc824142520 con 0x7fc82413f710 2024-03-25T18:36:18.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.706+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214988715 0 0) 0x7fc8240b2a30 con 0x7fc8240a98f0 2024-03-25T18:36:18.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.706+0000 7fc822ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:18.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.706+0000 7fc822ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:18.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.706+0000 7fc822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:18.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.706+0000 7fc822ffd640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:36:18.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.706+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8240b28a0 con 0x7fc8240a98f0 2024-03-25T18:36:18.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3106918783 0 0) 0x7fc80c003520 con 0x7fc8241430c0 2024-03-25T18:36:18.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 >> v1:172.21.15.102:6790/0 conn(0x7fc8240a98f0 legacy=0x7fc8240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:18.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 >> v1:172.21.15.146:6789/0 conn(0x7fc82413f710 legacy=0x7fc8241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:18.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:18.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 1 monclient: found mon.noname-a 2024-03-25T18:36:18.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8240b44c0 con 0x7fc8241430c0 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc81c001a10 con 0x7fc8241430c0 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc82a2e6640 5 monclient: authenticate success, global_id 4689 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc82a2e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc822ffd640 10 monclient: dump: 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.714 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:18.715 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:18.715 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:18.715 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:18.715 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:18.715 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:18.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.712+0000 7fc82a2e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:18.739 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.739+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc81c001b70 con 0x7fc8241430c0 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.739+0000 7fc822ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.739+0000 7fc822ffd640 1 -- 172.21.15.102:0/2876779981 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc81c001f80 con 0x7fc8241430c0 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.739+0000 7fc82a2e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.739+0000 7fc82a2e6640 4 set_mon_vals no callback set 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.739+0000 7fc822ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.739+0000 7fc822ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.739+0000 7fc822ffd640 10 monclient: dump: 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:18.740 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:18.741 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.740+0000 7fc82a2e6640 10 monclient: shutdown 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.740+0000 7fc82a2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.740+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2876779981 >> v1:172.21.15.102:6789/0 conn(0x7fc8241430c0 legacy=0x7fc82414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.740+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2876779981 shutdown_connections 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.740+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2876779981 >> 172.21.15.102:0/2876779981 conn(0x7fc8240b7ea0 msgr2=0x7fc8240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.740+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2876779981 shutdown_connections 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.740+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2876779981 wait complete. 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 10 monclient: build_initial_monmap 2024-03-25T18:36:18.742 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 10 monclient: monmap: 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 1 librados: starting msgr at 2024-03-25T18:36:18.743 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 1 librados: starting objecter 2024-03-25T18:36:18.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 1 Processor -- start 2024-03-25T18:36:18.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 1 -- start start 2024-03-25T18:36:18.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 1 librados: setting wanted keys 2024-03-25T18:36:18.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 1 librados: calling monclient init 2024-03-25T18:36:18.744 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.741+0000 7fc82a2e6640 10 monclient: init 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 10 monclient(hunting): picked mon.noname-b con 0x7fc8240a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 10 monclient(hunting): picked mon.noname-a con 0x7fc82413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 10 monclient(hunting): picked mon.noname-c con 0x7fc8241430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8240b7c30 con 0x7fc82413f710 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc824137520 con 0x7fc8240a98f0 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8241d7ba0 con 0x7fc8241430c0 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:18.745 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.742+0000 7fc82a2e6640 10 monclient(hunting): authenticate will time out at 8010.030539s 2024-03-25T18:36:18.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.749+0000 7fc823fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc8240a98f0 0x7fc8241d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:49162/0 (socket says 172.21.15.102:49162) 2024-03-25T18:36:18.749 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.749+0000 7fc823fff640 1 -- 172.21.15.102:0/2396737827 learned_addr learned my addr 172.21.15.102:0/2396737827 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:18.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.750+0000 7fc820ff9640 10 client.?.objecter ms_handle_connect 0x7fc8240a98f0 2024-03-25T18:36:18.751 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.750+0000 7fc820ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:18.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.757+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545281963 0 0) 0x7fc824137520 con 0x7fc8240a98f0 2024-03-25T18:36:18.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.757+0000 7fc820ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:18.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.757+0000 7fc820ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:18.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.757+0000 7fc820ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:18.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.757+0000 7fc820ff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:36:18.758 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.757+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc810004eb0 con 0x7fc8240a98f0 2024-03-25T18:36:18.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.758+0000 7fc820ff9640 10 client.?.objecter ms_handle_connect 0x7fc82413f710 2024-03-25T18:36:18.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.758+0000 7fc820ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:18.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.759+0000 7fc820ff9640 10 client.?.objecter ms_handle_connect 0x7fc8241430c0 2024-03-25T18:36:18.759 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.759+0000 7fc820ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:18.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.770+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065270111 0 0) 0x7fc8241d7ba0 con 0x7fc8241430c0 2024-03-25T18:36:18.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.770+0000 7fc820ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:18.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.770+0000 7fc820ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:18.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.770+0000 7fc820ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:18.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.770+0000 7fc820ff9640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:36:18.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.770+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc824137520 con 0x7fc8241430c0 2024-03-25T18:36:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.776+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400885574 0 0) 0x7fc8240b7c30 con 0x7fc82413f710 2024-03-25T18:36:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.776+0000 7fc820ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.776+0000 7fc820ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.776+0000 7fc820ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.776+0000 7fc820ff9640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:36:18.777 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.776+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8241d7ba0 con 0x7fc82413f710 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.786+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1578491464 0 0) 0x7fc824137520 con 0x7fc8241430c0 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.786+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8240b7c30 con 0x7fc8241430c0 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.786+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc81c001e80 con 0x7fc8241430c0 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.786+0000 7fc820ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.786+0000 7fc820ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.786+0000 7fc820ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.787 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.788 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:18.788 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:18.788 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:18.788 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:18.788 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:18.788 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:18.788 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.786+0000 7fc820ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.794+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1727414256 0 0) 0x7fc8241d7ba0 con 0x7fc82413f710 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.794+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc824137520 con 0x7fc82413f710 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.794+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc818001930 con 0x7fc82413f710 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.794+0000 7fc820ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.794+0000 7fc820ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.794+0000 7fc820ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.795 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:18.796 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:18.796 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:18.796 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:18.796 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:18.796 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:18.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.796+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3082581163 0 0) 0x7fc810004eb0 con 0x7fc8240a98f0 2024-03-25T18:36:18.796 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.796+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8241d7ba0 con 0x7fc8240a98f0 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.796+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc814001a10 con 0x7fc8240a98f0 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.796+0000 7fc820ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.796+0000 7fc820ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.796+0000 7fc820ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:18.797 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:18.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.804+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3794655661 0 0) 0x7fc824137520 con 0x7fc82413f710 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.804+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 >> v1:172.21.15.102:6790/0 conn(0x7fc8241430c0 legacy=0x7fc8241e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.804+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 >> v1:172.21.15.146:6789/0 conn(0x7fc8240a98f0 legacy=0x7fc8241d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc820ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc820ff9640 1 monclient: found mon.a 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc820ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc820ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc824136950 con 0x7fc82413f710 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc820ff9640 10 monclient: _finish_auth 0 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc820ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc820ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 5 monclient: authenticate success, global_id 4692 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 10 monclient: _renew_subs 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc824136ba0 con 0x7fc82413f710 2024-03-25T18:36:18.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:18.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 10 monclient: _renew_subs 2024-03-25T18:36:18.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:18.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc824137010 con 0x7fc82413f710 2024-03-25T18:36:18.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.805+0000 7fc82a2e6640 1 librados: init done 2024-03-25T18:36:18.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.806+0000 7fc82a2e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:18.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.806+0000 7fc82a2e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:18.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.806+0000 7fc82a2e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:18.807 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:18.806+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc824074cc0 con 0x7fc82413f710 2024-03-25T18:36:19.115 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.115+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8180035f0 con 0x7fc82413f710 2024-03-25T18:36:19.115 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.115+0000 7fc820ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:19.115 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.115+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc818001d80 con 0x7fc82413f710 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.115+0000 7fc820ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.115+0000 7fc820ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.115+0000 7fc820ff9640 10 monclient: dump: 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:19.116 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.115+0000 7fc8227fc640 4 set_mon_vals no callback set 2024-03-25T18:36:19.117 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.115+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fc818019830 con 0x7fc82413f710 2024-03-25T18:36:19.117 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.116+0000 7fc823fff640 10 monclient: get_auth_request con 0x7fc81006e110 auth_method 0 2024-03-25T18:36:19.117 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.116+0000 7fc820ff9640 10 client.4692.objecter ms_handle_connect 0x7fc81006e110 2024-03-25T18:36:19.122 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.121+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7fc818053520 con 0x7fc82413f710 2024-03-25T18:36:19.123 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.122+0000 7fc820ff9640 10 client.4692.objecter ms_dispatch 0x7fc824000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:19.123 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.122+0000 7fc820ff9640 3 client.4692.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:19.123 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.122+0000 7fc820ff9640 3 client.4692.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:19.123 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.122+0000 7fc820ff9640 20 client.4692.objecter dump_active .. 0 homeless 2024-03-25T18:36:19.123 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.122+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc81804ee60 con 0x7fc82413f710 2024-03-25T18:36:19.123 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.122+0000 7fc820ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:19.123 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.122+0000 7fc820ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:19.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.351+0000 7fc82a2e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:19.351 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.351+0000 7fc82a2e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:19.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.351+0000 7fc82a2e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:19.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.351+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc824137760 con 0x7fc82413f710 2024-03-25T18:36:19.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.376+0000 7fc820ff9640 1 -- 172.21.15.102:0/2396737827 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7fc824137760 con 0x7fc82413f710 2024-03-25T18:36:19.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.376+0000 7fc820ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:19.377 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.376+0000 7fc820ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:19.377 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:36:19.377 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:19.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 10 librados: watch_flush enter 2024-03-25T18:36:19.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 10 librados: watch_flush exit 2024-03-25T18:36:19.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 20 client.4692.objecter shutdown clearing up homeless session... 2024-03-25T18:36:19.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 10 client.4692.objecter successfully canceled tick 2024-03-25T18:36:19.379 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 >> v1:172.21.15.146:6800/3848252714 conn(0x7fc81006e110 legacy=0x7fc8100705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 10 monclient: shutdown 2024-03-25T18:36:19.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:19.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 >> v1:172.21.15.102:6789/0 conn(0x7fc82413f710 legacy=0x7fc824133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:19.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 shutdown_connections 2024-03-25T18:36:19.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 >> 172.21.15.102:0/2396737827 conn(0x7fc8240b7ea0 msgr2=0x7fc82413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:19.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 shutdown_connections 2024-03-25T18:36:19.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 1 -- 172.21.15.102:0/2396737827 wait complete. 2024-03-25T18:36:19.380 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:19.378+0000 7fc82a2e6640 1 librados: shutdown 2024-03-25T18:36:19.390 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:26.197 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:26.301 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.300+0000 7f4f6e4f6640 10 monclient: get_monmap_and_config 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.300+0000 7f4f6e4f6640 10 monclient: build_initial_monmap 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.300+0000 7f4f6e4f6640 10 monclient: monmap: 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:36:26.302103+0000 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:36:26.302103+0000 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:36:26.302 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:26.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.301+0000 7f4f6e4f6640 1 Processor -- start 2024-03-25T18:36:26.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.301+0000 7f4f6e4f6640 1 -- start start 2024-03-25T18:36:26.303 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.301+0000 7f4f6e4f6640 10 monclient: init 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f680b4d60 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f680b1730 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f6814be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680af940 con 0x7f4f6814be90 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680b0a70 con 0x7f4f680b1730 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f680b0d50 con 0x7f4f680b4d60 2024-03-25T18:36:26.304 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:26.305 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.302+0000 7f4f6e4f6640 10 monclient(hunting): authenticate will time out at 8017.591344s 2024-03-25T18:36:26.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.312+0000 7f4f6ca93640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4f6814be90 0x7f4f6814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46104/0 (socket says 172.21.15.102:46104) 2024-03-25T18:36:26.313 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.312+0000 7f4f6ca93640 1 -- 172.21.15.102:0/308080676 learned_addr learned my addr 172.21.15.102:0/308080676 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:26.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.321+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206280340 0 0) 0x7f4f680af940 con 0x7f4f6814be90 2024-03-25T18:36:26.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.321+0000 7f4f66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.321+0000 7f4f66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.321+0000 7f4f66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.321+0000 7f4f66ffd640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:36:26.323 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.321+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f50003520 con 0x7f4f6814be90 2024-03-25T18:36:26.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.328+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859073843 0 0) 0x7f4f680b0a70 con 0x7f4f680b1730 2024-03-25T18:36:26.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.328+0000 7f4f66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.330 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.328+0000 7f4f66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.328+0000 7f4f66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.328+0000 7f4f66ffd640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:36:26.331 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.328+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f680af940 con 0x7f4f680b1730 2024-03-25T18:36:26.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.331+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122960052 0 0) 0x7f4f680b0d50 con 0x7f4f680b4d60 2024-03-25T18:36:26.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.331+0000 7f4f66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.331+0000 7f4f66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.331+0000 7f4f66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.331+0000 7f4f66ffd640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:36:26.333 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.331+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f680b0a70 con 0x7f4f680b4d60 2024-03-25T18:36:26.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1930431384 0 0) 0x7f4f50003520 con 0x7f4f6814be90 2024-03-25T18:36:26.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 >> v1:172.21.15.102:6790/0 conn(0x7f4f680b4d60 legacy=0x7f4f6813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:26.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 >> v1:172.21.15.146:6789/0 conn(0x7f4f680b1730 legacy=0x7f4f680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:26.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 1 monclient: found mon.noname-a 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f680b1030 con 0x7f4f6814be90 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4f60001a10 con 0x7f4f6814be90 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f6e4f6640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f6e4f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f66ffd640 10 monclient: dump: 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:26.342 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:26.343 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.339+0000 7f4f6e4f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:26.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.363+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f60001b70 con 0x7f4f6814be90 2024-03-25T18:36:26.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.363+0000 7f4f66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:26.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.363+0000 7f4f66ffd640 1 -- 172.21.15.102:0/308080676 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4f60001f80 con 0x7f4f6814be90 2024-03-25T18:36:26.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.363+0000 7f4f6e4f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:26.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.363+0000 7f4f6e4f6640 4 set_mon_vals no callback set 2024-03-25T18:36:26.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.363+0000 7f4f66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:26.365 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.363+0000 7f4f66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:26.366 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.363+0000 7f4f66ffd640 10 monclient: dump: 2024-03-25T18:36:26.366 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:26.366 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:26.366 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.366 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.366 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:26.366 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:26.367 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:26.367 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:26.367 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:26.367 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.364+0000 7f4f6e4f6640 10 monclient: shutdown 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.364+0000 7f4f6e4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.364+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/308080676 >> v1:172.21.15.102:6789/0 conn(0x7f4f6814be90 legacy=0x7f4f6814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.364+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/308080676 shutdown_connections 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.364+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/308080676 >> 172.21.15.102:0/308080676 conn(0x7f4f680617a0 msgr2=0x7f4f68061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.364+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/308080676 shutdown_connections 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.364+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/308080676 wait complete. 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.365+0000 7f4f6e4f6640 10 monclient: build_initial_monmap 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.365+0000 7f4f6e4f6640 10 monclient: monmap: 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:26.370 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.365+0000 7f4f6e4f6640 1 librados: starting msgr at 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.365+0000 7f4f6e4f6640 1 librados: starting objecter 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.365+0000 7f4f6e4f6640 1 Processor -- start 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 1 -- start start 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 1 librados: setting wanted keys 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 1 librados: calling monclient init 2024-03-25T18:36:26.371 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 10 monclient: init 2024-03-25T18:36:26.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:26.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:36:26.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f680b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:26.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f680b4d60 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:26.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f6814be90 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:26.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f681d86f0 con 0x7f4f6814be90 2024-03-25T18:36:26.372 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6813b4f0 con 0x7f4f680b4d60 2024-03-25T18:36:26.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f681d7eb0 con 0x7f4f680b1730 2024-03-25T18:36:26.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:26.373 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.366+0000 7f4f6e4f6640 10 monclient(hunting): authenticate will time out at 8017.655678s 2024-03-25T18:36:26.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.379+0000 7f4f67fff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f4f680b1730 0x7f4f681d77a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:42318/0 (socket says 172.21.15.102:42318) 2024-03-25T18:36:26.381 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.379+0000 7f4f67fff640 1 -- 172.21.15.102:0/3126715060 learned_addr learned my addr 172.21.15.102:0/3126715060 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:26.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.385+0000 7f4f64ff9640 10 client.?.objecter ms_handle_connect 0x7f4f680b1730 2024-03-25T18:36:26.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.385+0000 7f4f64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:26.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.385+0000 7f4f64ff9640 10 client.?.objecter ms_handle_connect 0x7f4f6814be90 2024-03-25T18:36:26.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.385+0000 7f4f64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:26.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.385+0000 7f4f64ff9640 10 client.?.objecter ms_handle_connect 0x7f4f680b4d60 2024-03-25T18:36:26.387 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.385+0000 7f4f64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:26.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590097693 0 0) 0x7f4f680b1030 con 0x7f4f6814be90 2024-03-25T18:36:26.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:36:26.397 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f54004eb0 con 0x7f4f6814be90 2024-03-25T18:36:26.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193321696 0 0) 0x7f4f6813b4f0 con 0x7f4f680b4d60 2024-03-25T18:36:26.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:36:26.398 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.395+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f680b1030 con 0x7f4f680b4d60 2024-03-25T18:36:26.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.400+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538022842 0 0) 0x7f4f681d7eb0 con 0x7f4f680b1730 2024-03-25T18:36:26.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.400+0000 7f4f64ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:26.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.400+0000 7f4f64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:26.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.400+0000 7f4f64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:26.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.400+0000 7f4f64ff9640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:36:26.401 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.400+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f6813b4f0 con 0x7f4f680b1730 2024-03-25T18:36:26.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.407+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1686322500 0 0) 0x7f4f680b1030 con 0x7f4f680b4d60 2024-03-25T18:36:26.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.407+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f681d7eb0 con 0x7f4f680b4d60 2024-03-25T18:36:26.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.407+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4f5c001970 con 0x7f4f680b4d60 2024-03-25T18:36:26.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.407+0000 7f4f64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:26.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.407+0000 7f4f64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:26.409 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.407+0000 7f4f64ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:26.410 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.407+0000 7f4f64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.418+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 476785162 0 0) 0x7f4f54004eb0 con 0x7f4f6814be90 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.418+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f680b1030 con 0x7f4f6814be90 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.418+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4f60001ce0 con 0x7f4f6814be90 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.418+0000 7f4f64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.418+0000 7f4f64ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.418+0000 7f4f64ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:26.420 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.421 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.421 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:26.421 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:26.421 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:26.421 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:26.421 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:26.421 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 58799098 0 0) 0x7f4f680b1030 con 0x7f4f6814be90 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 >> v1:172.21.15.102:6790/0 conn(0x7f4f680b1730 legacy=0x7f4f681d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 >> v1:172.21.15.146:6789/0 conn(0x7f4f680b4d60 legacy=0x7f4f68136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 1 monclient: found mon.a 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f681d8040 con 0x7f4f6814be90 2024-03-25T18:36:26.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 10 monclient: _finish_auth 0 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 5 monclient: authenticate success, global_id 4698 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 10 monclient: _renew_subs 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6813ba70 con 0x7f4f6814be90 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 10 client.4698.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 10 monclient: _renew_subs 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f681378d0 con 0x7f4f6814be90 2024-03-25T18:36:26.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.427+0000 7f4f6e4f6640 1 librados: init done 2024-03-25T18:36:26.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.428+0000 7f4f6e4f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:26.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.428+0000 7f4f6e4f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:26.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.428+0000 7f4f6e4f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:26.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.428+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6813b730 con 0x7f4f6814be90 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.753+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f60004b60 con 0x7f4f6814be90 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.753+0000 7f4f64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.753+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f4f60004cd0 con 0x7f4f6814be90 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.753+0000 7f4f64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.753+0000 7f4f64ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.753+0000 7f4f64ff9640 10 monclient: dump: 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:26.755 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.753+0000 7f4f667fc640 4 set_mon_vals no callback set 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.754+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f4f6001aca0 con 0x7f4f6814be90 2024-03-25T18:36:26.756 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.754+0000 7f4f67fff640 10 monclient: get_auth_request con 0x7f4f5406dfb0 auth_method 0 2024-03-25T18:36:26.757 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.755+0000 7f4f64ff9640 10 client.4698.objecter ms_handle_connect 0x7f4f5406dfb0 2024-03-25T18:36:26.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.760+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7f4f600546e0 con 0x7f4f6814be90 2024-03-25T18:36:26.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.760+0000 7f4f64ff9640 10 client.4698.objecter ms_dispatch 0x7f4f68000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:26.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.760+0000 7f4f64ff9640 3 client.4698.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:26.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.760+0000 7f4f64ff9640 3 client.4698.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:26.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.760+0000 7f4f64ff9640 20 client.4698.objecter dump_active .. 0 homeless 2024-03-25T18:36:26.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.760+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f60050020 con 0x7f4f6814be90 2024-03-25T18:36:26.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.760+0000 7f4f64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:26.762 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:26.760+0000 7f4f64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.011+0000 7f4f6e4f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.011+0000 7f4f6e4f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:27.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.011+0000 7f4f6e4f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:27.013 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.011+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f680627b0 con 0x7f4f6814be90 2024-03-25T18:36:27.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.049+0000 7f4f64ff9640 1 -- 172.21.15.102:0/3126715060 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7f4f60052120 con 0x7f4f6814be90 2024-03-25T18:36:27.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.049+0000 7f4f64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:27.051 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.049+0000 7f4f64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:27.051 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:36:27.051 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:27.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 10 librados: watch_flush enter 2024-03-25T18:36:27.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 10 librados: watch_flush exit 2024-03-25T18:36:27.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 20 client.4698.objecter shutdown clearing up homeless session... 2024-03-25T18:36:27.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 10 client.4698.objecter successfully canceled tick 2024-03-25T18:36:27.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 >> v1:172.21.15.146:6800/3848252714 conn(0x7f4f5406dfb0 legacy=0x7f4f54070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:27.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 10 monclient: shutdown 2024-03-25T18:36:27.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:27.054 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 >> v1:172.21.15.102:6789/0 conn(0x7f4f6814be90 legacy=0x7f4f681e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:27.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 shutdown_connections 2024-03-25T18:36:27.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 >> 172.21.15.102:0/3126715060 conn(0x7f4f680617a0 msgr2=0x7f4f6813c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:27.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 shutdown_connections 2024-03-25T18:36:27.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 1 -- 172.21.15.102:0/3126715060 wait complete. 2024-03-25T18:36:27.055 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:27.052+0000 7f4f6e4f6640 1 librados: shutdown 2024-03-25T18:36:27.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:30.091 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:46.327 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.140 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:46.375 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.167 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:10:46.403 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.191 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:46.405 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.204 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:46.440 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.209 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:46.422 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.215 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:46.451 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.234 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:10:46.448 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:30.253 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:10:46.489 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:33.871 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:33.986 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.985+0000 7f43e94ff640 10 monclient: get_monmap_and_config 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.985+0000 7f43e94ff640 10 monclient: build_initial_monmap 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.985+0000 7f43e94ff640 10 monclient: monmap: 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:36:33.987101+0000 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:36:33.987101+0000 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:36:33.987 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:33.988 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 1 Processor -- start 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 1 -- start start 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 10 monclient: init 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 10 monclient(hunting): picked mon.noname-a con 0x7f43e40a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 10 monclient(hunting): picked mon.noname-b con 0x7f43e413f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 10 monclient(hunting): picked mon.noname-c con 0x7f43e41430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e4142520 con 0x7f43e40a98f0 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40b28a0 con 0x7f43e41430c0 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40b2a30 con 0x7f43e413f710 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:33.989 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.986+0000 7f43e94ff640 10 monclient(hunting): authenticate will time out at 8025.275663s 2024-03-25T18:36:33.997 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.995+0000 7f43e37fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f43e41430c0 0x7f43e414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:43510/0 (socket says 172.21.15.102:43510) 2024-03-25T18:36:33.997 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:33.995+0000 7f43e37fe640 1 -- 172.21.15.102:0/171286161 learned_addr learned my addr 172.21.15.102:0/171286161 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:34.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.001+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904643289 0 0) 0x7f43e40b28a0 con 0x7f43e41430c0 2024-03-25T18:36:34.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.001+0000 7f43e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.001+0000 7f43e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.001+0000 7f43e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.001+0000 7f43e1ffb640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:36:34.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.001+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc003520 con 0x7f43e41430c0 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1723880038 0 0) 0x7f43cc003520 con 0x7f43e41430c0 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 >> v1:172.21.15.102:6790/0 conn(0x7f43e413f710 legacy=0x7f43e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 >> v1:172.21.15.102:6789/0 conn(0x7f43e40a98f0 legacy=0x7f43e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e40b44c0 con 0x7f43e41430c0 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e94ff640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:36:34.012 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.010+0000 7f43e94ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:34.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f43dc001a10 con 0x7f43e41430c0 2024-03-25T18:36:34.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.016 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 10 monclient: dump: 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43dc001ce0 con 0x7f43e41430c0 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e94ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 1 -- 172.21.15.102:0/171286161 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f43dc002070 con 0x7f43e41430c0 2024-03-25T18:36:34.017 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e94ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e94ff640 4 set_mon_vals no callback set 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.015+0000 7f43e1ffb640 10 monclient: dump: 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:34.018 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:34.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.016+0000 7f43e94ff640 10 monclient: shutdown 2024-03-25T18:36:34.020 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.016+0000 7f43e94ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.016+0000 7f43e94ff640 1 -- 172.21.15.102:0/171286161 >> v1:172.21.15.146:6789/0 conn(0x7f43e41430c0 legacy=0x7f43e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.016+0000 7f43e94ff640 1 -- 172.21.15.102:0/171286161 shutdown_connections 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.016+0000 7f43e94ff640 1 -- 172.21.15.102:0/171286161 >> 172.21.15.102:0/171286161 conn(0x7f43e40b7ea0 msgr2=0x7f43e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.016+0000 7f43e94ff640 1 -- 172.21.15.102:0/171286161 shutdown_connections 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.016+0000 7f43e94ff640 1 -- 172.21.15.102:0/171286161 wait complete. 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.017+0000 7f43e94ff640 10 monclient: build_initial_monmap 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.017+0000 7f43e94ff640 10 monclient: monmap: 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:34.021 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.017+0000 7f43e94ff640 1 librados: starting msgr at 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.017+0000 7f43e94ff640 1 librados: starting objecter 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 1 Processor -- start 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 1 -- start start 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 1 librados: setting wanted keys 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 1 librados: calling monclient init 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 10 monclient: init 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 10 monclient(hunting): picked mon.noname-c con 0x7f43e40a98f0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.018+0000 7f43e94ff640 10 monclient(hunting): picked mon.noname-a con 0x7f43e413f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.019+0000 7f43e94ff640 10 monclient(hunting): picked mon.noname-b con 0x7f43e41430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:34.022 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.019+0000 7f43e94ff640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40aee10 con 0x7f43e413f710 2024-03-25T18:36:34.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.019+0000 7f43e94ff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e4139a30 con 0x7f43e41430c0 2024-03-25T18:36:34.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.019+0000 7f43e94ff640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e40b0b00 con 0x7f43e40a98f0 2024-03-25T18:36:34.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.019+0000 7f43e94ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:34.023 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.019+0000 7f43e94ff640 10 monclient(hunting): authenticate will time out at 8025.308004s 2024-03-25T18:36:34.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.029+0000 7f43e2ffd640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f43e40a98f0 0x7f43e40aefd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:52986/0 (socket says 172.21.15.102:52986) 2024-03-25T18:36:34.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.029+0000 7f43e2ffd640 1 -- 172.21.15.102:0/770398366 learned_addr learned my addr 172.21.15.102:0/770398366 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:34.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.031+0000 7f43cb7fe640 10 client.?.objecter ms_handle_connect 0x7f43e40a98f0 2024-03-25T18:36:34.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.031+0000 7f43cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:34.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.036+0000 7f43cb7fe640 10 client.?.objecter ms_handle_connect 0x7f43e413f710 2024-03-25T18:36:34.037 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.036+0000 7f43cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:34.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.037+0000 7f43cb7fe640 10 client.?.objecter ms_handle_connect 0x7f43e41430c0 2024-03-25T18:36:34.038 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.037+0000 7f43cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:34.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.040+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901645959 0 0) 0x7f43e40b0b00 con 0x7f43e40a98f0 2024-03-25T18:36:34.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.040+0000 7f43cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.040+0000 7f43cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.040+0000 7f43cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.040+0000 7f43cb7fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:36:34.042 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.040+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d0004eb0 con 0x7f43e40a98f0 2024-03-25T18:36:34.046 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.045+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574966525 0 0) 0x7f43d8001090 con 0x7f43e413f710 2024-03-25T18:36:34.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.045+0000 7f43cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.045+0000 7f43cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.045+0000 7f43cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.045+0000 7f43cb7fe640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:36:34.047 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.045+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e40b0b00 con 0x7f43e413f710 2024-03-25T18:36:34.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.047+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335100912 0 0) 0x7f43e4139a30 con 0x7f43e41430c0 2024-03-25T18:36:34.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.047+0000 7f43cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:36:34.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.047+0000 7f43cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:34.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.047+0000 7f43cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:34.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.047+0000 7f43cb7fe640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:36:34.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.047+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d8001090 con 0x7f43e41430c0 2024-03-25T18:36:34.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.059+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1307865312 0 0) 0x7f43d0004eb0 con 0x7f43e40a98f0 2024-03-25T18:36:34.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.060+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e4139a30 con 0x7f43e40a98f0 2024-03-25T18:36:34.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.060+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f43d40019f0 con 0x7f43e40a98f0 2024-03-25T18:36:34.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.060+0000 7f43cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.060+0000 7f43cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:34.061 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.060+0000 7f43cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:34.062 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.060+0000 7f43cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:34.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.068+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2450851462 0 0) 0x7f43d8001090 con 0x7f43e41430c0 2024-03-25T18:36:34.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.068+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43d0004eb0 con 0x7f43e41430c0 2024-03-25T18:36:34.069 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.068+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f43dc001830 con 0x7f43e41430c0 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.068+0000 7f43cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.068+0000 7f43cb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.068+0000 7f43cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:34.070 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.069+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2736412730 0 0) 0x7f43e40b0b00 con 0x7f43e413f710 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.069+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43d8001090 con 0x7f43e413f710 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.069+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f43d8001800 con 0x7f43e413f710 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.069+0000 7f43cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.069+0000 7f43cb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.069+0000 7f43cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.071 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.072 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:34.072 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:34.072 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:34.072 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:34.072 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:34.087 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3222983502 0 0) 0x7f43d0004eb0 con 0x7f43e41430c0 2024-03-25T18:36:34.087 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 >> v1:172.21.15.102:6790/0 conn(0x7f43e40a98f0 legacy=0x7f43e40aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.087 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 >> v1:172.21.15.102:6789/0 conn(0x7f43e413f710 legacy=0x7f43e4138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.087 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 1 monclient: found mon.b 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e40ae8f0 con 0x7f43e41430c0 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 10 monclient: _renew_subs 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e40ae4a0 con 0x7f43e41430c0 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 10 monclient: _renew_subs 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:34.088 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e414e140 con 0x7f43e41430c0 2024-03-25T18:36:34.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.085+0000 7f43e94ff640 1 librados: init done 2024-03-25T18:36:34.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.086+0000 7f43e94ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:34.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.086+0000 7f43e94ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:34.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.086+0000 7f43e94ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:34.089 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.086+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e4074cc0 con 0x7f43e41430c0 2024-03-25T18:36:34.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.088+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43dc001a50 con 0x7f43e41430c0 2024-03-25T18:36:34.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.088+0000 7f43cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:34.090 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.088+0000 7f43e17fa640 4 set_mon_vals no callback set 2024-03-25T18:36:34.352 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.351+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f43dc004fb0 con 0x7f43e41430c0 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.351+0000 7f43cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.351+0000 7f43cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.351+0000 7f43cb7fe640 10 monclient: dump: 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:34.353 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:34.354 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.352+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f43dc01af80 con 0x7f43e41430c0 2024-03-25T18:36:34.354 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.353+0000 7f43e2ffd640 10 monclient: get_auth_request con 0x7f43d006e110 auth_method 0 2024-03-25T18:36:34.354 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.353+0000 7f43cb7fe640 10 client.4711.objecter ms_handle_connect 0x7f43d006e110 2024-03-25T18:36:34.356 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.354+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7f43dc054a90 con 0x7f43e41430c0 2024-03-25T18:36:34.356 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.354+0000 7f43cb7fe640 10 client.4711.objecter ms_dispatch 0x7f43e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:34.356 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.354+0000 7f43cb7fe640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:34.356 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.354+0000 7f43cb7fe640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:34.356 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.354+0000 7f43cb7fe640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T18:36:34.360 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.359+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43dc0503d0 con 0x7f43e41430c0 2024-03-25T18:36:34.360 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.359+0000 7f43cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:34.361 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.359+0000 7f43cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:34.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.609+0000 7f43e94ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:34.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.609+0000 7f43e94ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:34.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.609+0000 7f43e94ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:36:34.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.609+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43e40627b0 con 0x7f43e41430c0 2024-03-25T18:36:34.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.648+0000 7f43cb7fe640 1 -- 172.21.15.102:0/770398366 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7f43dc0524d0 con 0x7f43e41430c0 2024-03-25T18:36:34.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.649+0000 7f43cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:34.650 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.649+0000 7f43cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:34.651 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:36:34.651 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 10 librados: watch_flush enter 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 10 librados: watch_flush exit 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 10 client.4711.objecter successfully canceled tick 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 >> v1:172.21.15.146:6800/3848252714 conn(0x7f43d006e110 legacy=0x7f43d00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 10 monclient: shutdown 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:34.653 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 >> v1:172.21.15.146:6789/0 conn(0x7f43e41430c0 legacy=0x7f43e4139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:34.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 shutdown_connections 2024-03-25T18:36:34.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 >> 172.21.15.102:0/770398366 conn(0x7f43e40b7ea0 msgr2=0x7f43e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:34.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 shutdown_connections 2024-03-25T18:36:34.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 1 -- 172.21.15.102:0/770398366 wait complete. 2024-03-25T18:36:34.654 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:34.651+0000 7f43e94ff640 1 librados: shutdown 2024-03-25T18:36:34.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:41.471 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:41.582 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.581+0000 7f8bce88f640 10 monclient: get_monmap_and_config 2024-03-25T18:36:41.582 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.581+0000 7f8bce88f640 10 monclient: build_initial_monmap 2024-03-25T18:36:41.582 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.581+0000 7f8bce88f640 10 monclient: monmap: 2024-03-25T18:36:41.582 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:41.582 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:41.582 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:36:41.582326+0000 2024-03-25T18:36:41.582 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:36:41.582326+0000 2024-03-25T18:36:41.582 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:41.583 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:41.583 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:41.583 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:36:41.583 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:36:41.583 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:41.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.582+0000 7f8bce88f640 1 Processor -- start 2024-03-25T18:36:41.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.582+0000 7f8bce88f640 1 -- start start 2024-03-25T18:36:41.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.582+0000 7f8bce88f640 10 monclient: init 2024-03-25T18:36:41.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:41.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:41.584 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 10 monclient(hunting): picked mon.noname-a con 0x7f8bc80a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:41.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 10 monclient(hunting): picked mon.noname-c con 0x7f8bc813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:41.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 10 monclient(hunting): picked mon.noname-b con 0x7f8bc81430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:41.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc8142520 con 0x7f8bc80a98f0 2024-03-25T18:36:41.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80b28a0 con 0x7f8bc813f710 2024-03-25T18:36:41.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80b2a30 con 0x7f8bc81430c0 2024-03-25T18:36:41.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:41.585 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.583+0000 7f8bce88f640 10 monclient(hunting): authenticate will time out at 8032.871280s 2024-03-25T18:36:41.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.588+0000 7f8bc7fff640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8bc80a98f0 0x7f8bc80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36874/0 (socket says 172.21.15.102:36874) 2024-03-25T18:36:41.589 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.588+0000 7f8bc7fff640 1 -- 172.21.15.102:0/165929065 learned_addr learned my addr 172.21.15.102:0/165929065 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:41.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.602+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153637990 0 0) 0x7f8bc8142520 con 0x7f8bc80a98f0 2024-03-25T18:36:41.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.602+0000 7f8bc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:41.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.602+0000 7f8bc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:41.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.602+0000 7f8bc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:41.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.602+0000 7f8bc6ffd640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:36:41.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.602+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb0003520 con 0x7f8bc80a98f0 2024-03-25T18:36:41.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.604+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949491622 0 0) 0x7f8bc80b28a0 con 0x7f8bc813f710 2024-03-25T18:36:41.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.604+0000 7f8bc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:41.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.604+0000 7f8bc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:41.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.604+0000 7f8bc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:41.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.604+0000 7f8bc6ffd640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:36:41.605 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.604+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc8142520 con 0x7f8bc813f710 2024-03-25T18:36:41.613 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4279542695 0 0) 0x7f8bc8142520 con 0x7f8bc813f710 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 >> v1:172.21.15.102:6790/0 conn(0x7f8bc81430c0 legacy=0x7f8bc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 >> v1:172.21.15.102:6789/0 conn(0x7f8bc80a98f0 legacy=0x7f8bc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80b44c0 con 0x7f8bc813f710 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8bbc001670 con 0x7f8bc813f710 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:41.614 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bc6ffd640 10 monclient: dump: 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bce88f640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:36:41.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.613+0000 7f8bce88f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:41.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.636+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bbc0017d0 con 0x7f8bc813f710 2024-03-25T18:36:41.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.636+0000 7f8bc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:41.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.636+0000 7f8bc6ffd640 1 -- 172.21.15.102:0/165929065 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8bbc001be0 con 0x7f8bc813f710 2024-03-25T18:36:41.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.636+0000 7f8bce88f640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:41.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.636+0000 7f8bce88f640 4 set_mon_vals no callback set 2024-03-25T18:36:41.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.636+0000 7f8bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.636+0000 7f8bc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.636+0000 7f8bc6ffd640 10 monclient: dump: 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:41.638 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:41.639 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.637+0000 7f8bce88f640 10 monclient: shutdown 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.637+0000 7f8bce88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.637+0000 7f8bce88f640 1 -- 172.21.15.102:0/165929065 >> v1:172.21.15.146:6789/0 conn(0x7f8bc813f710 legacy=0x7f8bc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.637+0000 7f8bce88f640 1 -- 172.21.15.102:0/165929065 shutdown_connections 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.637+0000 7f8bce88f640 1 -- 172.21.15.102:0/165929065 >> 172.21.15.102:0/165929065 conn(0x7f8bc80b7ea0 msgr2=0x7f8bc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.637+0000 7f8bce88f640 1 -- 172.21.15.102:0/165929065 shutdown_connections 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.637+0000 7f8bce88f640 1 -- 172.21.15.102:0/165929065 wait complete. 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.638+0000 7f8bce88f640 10 monclient: build_initial_monmap 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.638+0000 7f8bce88f640 10 monclient: monmap: 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:41.641 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.638+0000 7f8bce88f640 1 librados: starting msgr at 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.638+0000 7f8bce88f640 1 librados: starting objecter 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.638+0000 7f8bce88f640 1 Processor -- start 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 1 -- start start 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 1 librados: setting wanted keys 2024-03-25T18:36:41.642 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 1 librados: calling monclient init 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 10 monclient: init 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 10 monclient(hunting): picked mon.noname-b con 0x7f8bc80a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 10 monclient(hunting): picked mon.noname-c con 0x7f8bc813f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 10 monclient(hunting): picked mon.noname-a con 0x7f8bc81430c0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc812dbe0 con 0x7f8bc81430c0 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc813a170 con 0x7f8bc80a98f0 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80b7bc0 con 0x7f8bc813f710 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:41.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.639+0000 7f8bce88f640 10 monclient(hunting): authenticate will time out at 8032.927732s 2024-03-25T18:36:41.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.647+0000 7f8bc7fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8bc80a98f0 0x7f8bc81d7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:50960/0 (socket says 172.21.15.102:50960) 2024-03-25T18:36:41.647 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.647+0000 7f8bc7fff640 1 -- 172.21.15.102:0/2219783031 learned_addr learned my addr 172.21.15.102:0/2219783031 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:41.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.648+0000 7f8bc4ff9640 10 client.?.objecter ms_handle_connect 0x7f8bc80a98f0 2024-03-25T18:36:41.648 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.648+0000 7f8bc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:41.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.648+0000 7f8bc4ff9640 10 client.?.objecter ms_handle_connect 0x7f8bc81430c0 2024-03-25T18:36:41.649 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.648+0000 7f8bc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:41.652 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.652+0000 7f8bc4ff9640 10 client.?.objecter ms_handle_connect 0x7f8bc813f710 2024-03-25T18:36:41.652 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.652+0000 7f8bc4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:41.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.656+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401484690 0 0) 0x7f8bc812dbe0 con 0x7f8bc81430c0 2024-03-25T18:36:41.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.656+0000 7f8bc4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:41.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.656+0000 7f8bc4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:41.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.656+0000 7f8bc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:41.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.656+0000 7f8bc4ff9640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:36:41.657 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.656+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb4004eb0 con 0x7f8bc81430c0 2024-03-25T18:36:41.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.657+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628611756 0 0) 0x7f8bc80b7bc0 con 0x7f8bc813f710 2024-03-25T18:36:41.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.657+0000 7f8bc4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:41.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.657+0000 7f8bc4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:41.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.657+0000 7f8bc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:41.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.657+0000 7f8bc4ff9640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:36:41.658 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.657+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc812dbe0 con 0x7f8bc813f710 2024-03-25T18:36:41.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.662+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485910047 0 0) 0x7f8bc813a170 con 0x7f8bc80a98f0 2024-03-25T18:36:41.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.662+0000 7f8bc4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:36:41.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.662+0000 7f8bc4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:41.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.662+0000 7f8bc4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:41.662 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.662+0000 7f8bc4ff9640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:36:41.663 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.662+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc80b7bc0 con 0x7f8bc80a98f0 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.667+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4180468392 0 0) 0x7f8bc812dbe0 con 0x7f8bc813f710 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.667+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bc813a170 con 0x7f8bc813f710 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.667+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8bbc001ae0 con 0x7f8bc813f710 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.667+0000 7f8bc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.667+0000 7f8bc4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.667+0000 7f8bc4ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:41.668 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:41.669 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:41.669 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:41.669 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:41.669 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:41.669 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.667+0000 7f8bc4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.669+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 727007398 0 0) 0x7f8bb4004eb0 con 0x7f8bc81430c0 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.669+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bc812dbe0 con 0x7f8bc81430c0 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.669+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8bc0001ad0 con 0x7f8bc81430c0 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.669+0000 7f8bc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.669+0000 7f8bc4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.669+0000 7f8bc4ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:41.670 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:41.671 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:41.671 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:41.671 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:41.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.673+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2851483904 0 0) 0x7f8bc80b7bc0 con 0x7f8bc80a98f0 2024-03-25T18:36:41.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.673+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bb4004eb0 con 0x7f8bc80a98f0 2024-03-25T18:36:41.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.673+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8bb8001950 con 0x7f8bc80a98f0 2024-03-25T18:36:41.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.673+0000 7f8bc4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:41.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.674+0000 7f8bc4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.674+0000 7f8bc4ff9640 10 monclient(hunting): dump: 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:41.675 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:41.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 549158613 0 0) 0x7f8bc813a170 con 0x7f8bc813f710 2024-03-25T18:36:41.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 >> v1:172.21.15.146:6789/0 conn(0x7f8bc80a98f0 legacy=0x7f8bc81d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:41.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 >> v1:172.21.15.102:6789/0 conn(0x7f8bc81430c0 legacy=0x7f8bc812d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:41.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 1 monclient: found mon.c 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8139d20 con 0x7f8bc813f710 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 5 monclient: authenticate success, global_id 4688 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 10 monclient: _renew_subs 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc8139f10 con 0x7f8bc813f710 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 10 client.4688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:41.677 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 10 monclient: _renew_subs 2024-03-25T18:36:41.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:41.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc80b7390 con 0x7f8bc813f710 2024-03-25T18:36:41.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.675+0000 7f8bce88f640 1 librados: init done 2024-03-25T18:36:41.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.676+0000 7f8bce88f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:41.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.676+0000 7f8bce88f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:41.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.676+0000 7f8bce88f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:41.679 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:41.676+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc8074cc0 con 0x7f8bc813f710 2024-03-25T18:36:42.002 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.002+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bbc001210 con 0x7f8bc813f710 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.002+0000 7f8bc4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.002+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8bbc003a40 con 0x7f8bc813f710 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.002+0000 7f8bc4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.002+0000 7f8bc4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.002+0000 7f8bc4ff9640 10 monclient: dump: 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:42.003 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:42.004 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:42.004 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:42.004 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:42.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.002+0000 7f8bc67fc640 4 set_mon_vals no callback set 2024-03-25T18:36:42.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.003+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f8bbc019a10 con 0x7f8bc813f710 2024-03-25T18:36:42.004 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.003+0000 7f8bc7fff640 10 monclient: get_auth_request con 0x7f8bb406e0f0 auth_method 0 2024-03-25T18:36:42.005 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.004+0000 7f8bc4ff9640 10 client.4688.objecter ms_handle_connect 0x7f8bb406e0f0 2024-03-25T18:36:42.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.007+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7f8bbc053660 con 0x7f8bc813f710 2024-03-25T18:36:42.008 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.007+0000 7f8bc4ff9640 10 client.4688.objecter ms_dispatch 0x7f8bc8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:42.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.007+0000 7f8bc4ff9640 3 client.4688.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:42.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.007+0000 7f8bc4ff9640 3 client.4688.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:42.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.008+0000 7f8bc4ff9640 20 client.4688.objecter dump_active .. 0 homeless 2024-03-25T18:36:42.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.008+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8bbc04efa0 con 0x7f8bc813f710 2024-03-25T18:36:42.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.008+0000 7f8bc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:42.009 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.008+0000 7f8bc4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:42.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.264+0000 7f8bce88f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:42.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.264+0000 7f8bce88f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:42.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.264+0000 7f8bce88f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:42.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.264+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bc80627b0 con 0x7f8bc813f710 2024-03-25T18:36:42.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.283+0000 7f8bc4ff9640 1 -- 172.21.15.102:0/2219783031 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7f8bbc0510a0 con 0x7f8bc813f710 2024-03-25T18:36:42.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.283+0000 7f8bc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:42.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.283+0000 7f8bc4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:42.284 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:36:42.285 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:42.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 10 librados: watch_flush enter 2024-03-25T18:36:42.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 10 librados: watch_flush exit 2024-03-25T18:36:42.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 20 client.4688.objecter shutdown clearing up homeless session... 2024-03-25T18:36:42.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 10 client.4688.objecter successfully canceled tick 2024-03-25T18:36:42.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 >> v1:172.21.15.146:6800/3848252714 conn(0x7f8bb406e0f0 legacy=0x7f8bb40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:42.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 10 monclient: shutdown 2024-03-25T18:36:42.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:42.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 >> v1:172.21.15.102:6790/0 conn(0x7f8bc813f710 legacy=0x7f8bc812cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:42.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 shutdown_connections 2024-03-25T18:36:42.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.286+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 >> 172.21.15.102:0/2219783031 conn(0x7f8bc80b7ea0 msgr2=0x7f8bc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:42.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.287+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 shutdown_connections 2024-03-25T18:36:42.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.287+0000 7f8bce88f640 1 -- 172.21.15.102:0/2219783031 wait complete. 2024-03-25T18:36:42.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:42.287+0000 7f8bce88f640 1 librados: shutdown 2024-03-25T18:36:42.301 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:49.107 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.226+0000 7f8a447a8640 10 monclient: get_monmap_and_config 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.226+0000 7f8a447a8640 10 monclient: build_initial_monmap 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.226+0000 7f8a447a8640 10 monclient: monmap: 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:36:49.226964+0000 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:36:49.226964+0000 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:49.227 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:36:49.228 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:36:49.228 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.226+0000 7f8a447a8640 1 Processor -- start 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.226+0000 7f8a447a8640 1 -- start start 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 10 monclient: init 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3c0a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3c13f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8a3c1430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c142520 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.229 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c0b28a0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c0b2a30 con 0x7f8a3c13f710 2024-03-25T18:36:49.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:49.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.227+0000 7f8a447a8640 10 monclient(hunting): authenticate will time out at 8040.515728s 2024-03-25T18:36:49.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.238+0000 7f8a42544640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8a3c0a98f0 0x7f8a3c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41766/0 (socket says 172.21.15.102:41766) 2024-03-25T18:36:49.239 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.238+0000 7f8a42544640 1 -- 172.21.15.102:0/2228703418 learned_addr learned my addr 172.21.15.102:0/2228703418 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:49.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.250+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435503457 0 0) 0x7f8a3c142520 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.250+0000 7f8a41542640 10 monclient(hunting): _negotiate 2024-03-25T18:36:49.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.250+0000 7f8a41542640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:49.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.250+0000 7f8a41542640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:49.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.250+0000 7f8a41542640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:36:49.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.250+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a28003520 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.255+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278592329 0 0) 0x7f8a3c0b28a0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.255+0000 7f8a41542640 10 monclient(hunting): _negotiate 2024-03-25T18:36:49.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.255+0000 7f8a41542640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:49.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.255+0000 7f8a41542640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:49.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.255+0000 7f8a41542640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:36:49.256 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.255+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c142520 con 0x7f8a3c1430c0 2024-03-25T18:36:49.268 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3063880104 0 0) 0x7f8a28003520 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 >> v1:172.21.15.102:6790/0 conn(0x7f8a3c13f710 legacy=0x7f8a3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:49.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 >> v1:172.21.15.146:6789/0 conn(0x7f8a3c1430c0 legacy=0x7f8a3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:49.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 10 monclient: _finish_hunting 0 2024-03-25T18:36:49.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 1 monclient: found mon.noname-a 2024-03-25T18:36:49.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:49.269 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:36:49.270 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c0b44c0 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.270 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 10 monclient: _finish_auth 0 2024-03-25T18:36:49.270 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 10 monclient: _check_auth_tickets 2024-03-25T18:36:49.270 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:49.270 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8a38001690 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.270 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:49.270 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a41542640 10 monclient: dump: 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:49.271 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:49.272 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:49.272 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a447a8640 5 monclient: authenticate success, global_id 4710 2024-03-25T18:36:49.272 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.268+0000 7f8a447a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:49.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.283+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a380017f0 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.283+0000 7f8a41542640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:49.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.283+0000 7f8a41542640 1 -- 172.21.15.102:0/2228703418 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8a38001c00 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.284 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.283+0000 7f8a447a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.283+0000 7f8a447a8640 4 set_mon_vals no callback set 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.283+0000 7f8a41542640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.283+0000 7f8a41542640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.283+0000 7f8a41542640 10 monclient: dump: 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:49.285 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:49.286 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:49.286 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:49.286 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:49.286 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:49.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.284+0000 7f8a447a8640 10 monclient: shutdown 2024-03-25T18:36:49.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.284+0000 7f8a447a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:49.288 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.284+0000 7f8a447a8640 1 -- 172.21.15.102:0/2228703418 >> v1:172.21.15.102:6789/0 conn(0x7f8a3c0a98f0 legacy=0x7f8a3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.284+0000 7f8a447a8640 1 -- 172.21.15.102:0/2228703418 shutdown_connections 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.284+0000 7f8a447a8640 1 -- 172.21.15.102:0/2228703418 >> 172.21.15.102:0/2228703418 conn(0x7f8a3c0b7ea0 msgr2=0x7f8a3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.285+0000 7f8a447a8640 1 -- 172.21.15.102:0/2228703418 shutdown_connections 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.285+0000 7f8a447a8640 1 -- 172.21.15.102:0/2228703418 wait complete. 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.285+0000 7f8a447a8640 10 monclient: build_initial_monmap 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.285+0000 7f8a447a8640 10 monclient: monmap: 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:36:49.289 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 librados: starting msgr at 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 librados: starting objecter 2024-03-25T18:36:49.290 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 Processor -- start 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 -- start start 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 librados: setting wanted keys 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 librados: calling monclient init 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 10 monclient: init 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3c0a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3c13f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8a3c1430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c132a30 con 0x7f8a3c13f710 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c138c80 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a3c138f60 con 0x7f8a3c1430c0 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:49.291 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.286+0000 7f8a447a8640 10 monclient(hunting): authenticate will time out at 8040.574773s 2024-03-25T18:36:49.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.293+0000 7f8a42d45640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f8a3c1430c0 0x7f8a3c132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:56870/0 (socket says 172.21.15.102:56870) 2024-03-25T18:36:49.294 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.293+0000 7f8a42d45640 1 -- 172.21.15.102:0/2031142779 learned_addr learned my addr 172.21.15.102:0/2031142779 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:49.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.294+0000 7f8a32ffd640 10 client.?.objecter ms_handle_connect 0x7f8a3c1430c0 2024-03-25T18:36:49.295 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.294+0000 7f8a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:49.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.295+0000 7f8a32ffd640 10 client.?.objecter ms_handle_connect 0x7f8a3c13f710 2024-03-25T18:36:49.296 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.295+0000 7f8a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:49.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.298+0000 7f8a32ffd640 10 client.?.objecter ms_handle_connect 0x7f8a3c0a98f0 2024-03-25T18:36:49.299 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.298+0000 7f8a32ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:49.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.317+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267280699 0 0) 0x7f8a3c138f60 con 0x7f8a3c1430c0 2024-03-25T18:36:49.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.317+0000 7f8a32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:49.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.317+0000 7f8a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:49.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.317+0000 7f8a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:49.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.317+0000 7f8a32ffd640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:36:49.318 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.317+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a20004eb0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.392+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1405721953 0 0) 0x7f8a20004eb0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.392+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a3c138f60 con 0x7f8a3c1430c0 2024-03-25T18:36:49.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.392+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8a34001c90 con 0x7f8a3c1430c0 2024-03-25T18:36:49.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.392+0000 7f8a32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:49.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.392+0000 7f8a32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:36:49.394 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.392+0000 7f8a32ffd640 10 monclient(hunting): dump: 2024-03-25T18:36:49.394 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:49.395 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.392+0000 7f8a32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:49.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.393+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237988483 0 0) 0x7f8a3c138c80 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.393+0000 7f8a32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:49.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.393+0000 7f8a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:49.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.393+0000 7f8a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:49.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.393+0000 7f8a32ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:36:49.396 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.393+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a20004eb0 con 0x7f8a3c0a98f0 2024-03-25T18:36:49.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 788384367 0 0) 0x7f8a3c138f60 con 0x7f8a3c1430c0 2024-03-25T18:36:49.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 >> v1:172.21.15.146:6789/0 conn(0x7f8a3c0a98f0 legacy=0x7f8a3c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:49.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 >> v1:172.21.15.102:6789/0 conn(0x7f8a3c13f710 legacy=0x7f8a3c131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:49.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:49.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 1 monclient: found mon.c 2024-03-25T18:36:49.403 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c139240 con 0x7f8a3c1430c0 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a447a8640 5 monclient: authenticate success, global_id 4694 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a447a8640 10 monclient: _renew_subs 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a447a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c136e50 con 0x7f8a3c1430c0 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a447a8640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a447a8640 10 monclient: _renew_subs 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a447a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:49.404 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.401+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c1389f0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.402+0000 7f8a447a8640 1 librados: init done 2024-03-25T18:36:49.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.403+0000 7f8a447a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:49.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.403+0000 7f8a447a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:49.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.403+0000 7f8a447a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:49.405 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.403+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3c0627b0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.687+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a34003990 con 0x7f8a3c1430c0 2024-03-25T18:36:49.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.687+0000 7f8a32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:49.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.687+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f8a34003d20 con 0x7f8a3c1430c0 2024-03-25T18:36:49.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.687+0000 7f8a32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:49.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.687+0000 7f8a32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:49.688 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.687+0000 7f8a32ffd640 10 monclient: dump: 2024-03-25T18:36:49.688 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:49.688 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:49.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.687+0000 7f8a40d41640 4 set_mon_vals no callback set 2024-03-25T18:36:49.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.688+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f8a34003f40 con 0x7f8a3c1430c0 2024-03-25T18:36:49.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.689+0000 7f8a42544640 10 monclient: get_auth_request con 0x7f8a2006dd90 auth_method 0 2024-03-25T18:36:49.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.689+0000 7f8a32ffd640 10 client.4694.objecter ms_handle_connect 0x7f8a2006dd90 2024-03-25T18:36:49.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.705+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7f8a34054ab0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.705+0000 7f8a32ffd640 10 client.4694.objecter ms_dispatch 0x7f8a3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:49.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.705+0000 7f8a32ffd640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:49.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.705+0000 7f8a32ffd640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:49.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.705+0000 7f8a32ffd640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T18:36:49.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.705+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a340505c0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.706 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.705+0000 7f8a32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:49.707 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.705+0000 7f8a32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:49.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.949+0000 7f8a447a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:49.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.949+0000 7f8a447a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:49.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.949+0000 7f8a447a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:36:49.950 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.949+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a3c13dd60 con 0x7f8a3c1430c0 2024-03-25T18:36:49.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.974+0000 7f8a32ffd640 1 -- 172.21.15.102:0/2031142779 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7f8a340524f0 con 0x7f8a3c1430c0 2024-03-25T18:36:49.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.974+0000 7f8a32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:49.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.974+0000 7f8a32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:49.975 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:36:49.975 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 10 librados: watch_flush enter 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 10 librados: watch_flush exit 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 10 client.4694.objecter successfully canceled tick 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 >> v1:172.21.15.146:6800/3848252714 conn(0x7f8a2006dd90 legacy=0x7f8a20070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 10 monclient: shutdown 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 >> v1:172.21.15.102:6790/0 conn(0x7f8a3c1430c0 legacy=0x7f8a3c132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:49.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 shutdown_connections 2024-03-25T18:36:49.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 >> 172.21.15.102:0/2031142779 conn(0x7f8a3c0b7ea0 msgr2=0x7f8a3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:49.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 shutdown_connections 2024-03-25T18:36:49.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.976+0000 7f8a447a8640 1 -- 172.21.15.102:0/2031142779 wait complete. 2024-03-25T18:36:49.979 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:49.977+0000 7f8a447a8640 1 librados: shutdown 2024-03-25T18:36:49.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:36:50.209 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:06.445 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:50.211 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:06.447 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:50.251 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:06.487 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:50.278 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:06.514 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:50.280 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:06.516 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:50.283 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:06.496 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:50.289 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:06.502 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:50.307 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:06.543 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:50.332 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:06.545 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:36:56.798 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.913+0000 7f1c78ad9640 10 monclient: get_monmap_and_config 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.913+0000 7f1c78ad9640 10 monclient: build_initial_monmap 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.913+0000 7f1c78ad9640 10 monclient: monmap: 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:36:56.914165+0000 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:36:56.914165+0000 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:56.914 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:56.915 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:56.915 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:36:56.915 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:36:56.915 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:56.916 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.914+0000 7f1c78ad9640 1 Processor -- start 2024-03-25T18:36:56.916 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.914+0000 7f1c78ad9640 1 -- start start 2024-03-25T18:36:56.916 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.914+0000 7f1c78ad9640 10 monclient: init 2024-03-25T18:36:56.916 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:56.916 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:36:56.916 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 10 monclient(hunting): picked mon.noname-a con 0x7f1c740705b0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:56.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 10 monclient(hunting): picked mon.noname-c con 0x7f1c741402d0 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:56.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 10 monclient(hunting): picked mon.noname-b con 0x7f1c74149ae0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:56.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c740718c0 con 0x7f1c740705b0 2024-03-25T18:36:56.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c740714c0 con 0x7f1c741402d0 2024-03-25T18:36:56.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c74072b50 con 0x7f1c74149ae0 2024-03-25T18:36:56.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:56.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.915+0000 7f1c78ad9640 10 monclient(hunting): authenticate will time out at 8048.203352s 2024-03-25T18:36:56.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.917+0000 7f1c72d9d640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f1c74149ae0 0x7f1c7414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:39078/0 (socket says 172.21.15.102:39078) 2024-03-25T18:36:56.918 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.917+0000 7f1c72d9d640 1 -- 172.21.15.102:0/1737699489 learned_addr learned my addr 172.21.15.102:0/1737699489 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:56.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.933+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438257762 0 0) 0x7f1c74072b50 con 0x7f1c74149ae0 2024-03-25T18:36:56.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.933+0000 7f1c7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:36:56.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.933+0000 7f1c7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:56.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.933+0000 7f1c7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:56.934 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.933+0000 7f1c7159a640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:36:56.935 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.933+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c5c003520 con 0x7f1c74149ae0 2024-03-25T18:36:56.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.935+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852469777 0 0) 0x7f1c740718c0 con 0x7f1c740705b0 2024-03-25T18:36:56.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.935+0000 7f1c7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:36:56.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.935+0000 7f1c7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:56.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.935+0000 7f1c7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:56.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.935+0000 7f1c7159a640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:36:56.936 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.935+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c74072b50 con 0x7f1c740705b0 2024-03-25T18:36:56.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.953+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105939384 0 0) 0x7f1c740714c0 con 0x7f1c741402d0 2024-03-25T18:36:56.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.953+0000 7f1c7159a640 10 monclient(hunting): _negotiate 2024-03-25T18:36:56.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.953+0000 7f1c7159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:56.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.953+0000 7f1c7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:56.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.953+0000 7f1c7159a640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:36:56.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.953+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c740718c0 con 0x7f1c741402d0 2024-03-25T18:36:56.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3470030350 0 0) 0x7f1c5c003520 con 0x7f1c74149ae0 2024-03-25T18:36:56.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 >> v1:172.21.15.146:6789/0 conn(0x7f1c741402d0 legacy=0x7f1c74140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:56.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 >> v1:172.21.15.102:6789/0 conn(0x7f1c740705b0 legacy=0x7f1c7413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:56.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 10 monclient: _finish_hunting 0 2024-03-25T18:36:56.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 1 monclient: found mon.noname-b 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7413dd10 con 0x7f1c74149ae0 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 10 monclient: _finish_auth 0 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 10 monclient: _check_auth_tickets 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1c6c001a10 con 0x7f1c74149ae0 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c7159a640 10 monclient: dump: 2024-03-25T18:36:56.957 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:56.958 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:56.958 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:56.958 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:56.958 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:56.958 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:56.959 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:56.959 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:56.959 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:56.959 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:56.959 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c78ad9640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:36:56.959 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.955+0000 7f1c78ad9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:36:56.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.967+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c6c001b70 con 0x7f1c74149ae0 2024-03-25T18:36:56.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.967+0000 7f1c7159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:56.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.967+0000 7f1c7159a640 1 -- 172.21.15.102:0/1737699489 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1c6c001f80 con 0x7f1c74149ae0 2024-03-25T18:36:56.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.967+0000 7f1c78ad9640 10 monclient: get_monmap_and_config success 2024-03-25T18:36:56.968 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.967+0000 7f1c78ad9640 4 set_mon_vals no callback set 2024-03-25T18:36:56.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.967+0000 7f1c7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:56.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.967+0000 7f1c7159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:56.969 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.967+0000 7f1c7159a640 10 monclient: dump: 2024-03-25T18:36:56.969 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:56.969 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:56.969 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:56.970 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:56.970 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:56.970 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:56.970 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:56.970 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:56.970 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:56.970 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:56.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.968+0000 7f1c78ad9640 10 monclient: shutdown 2024-03-25T18:36:56.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.968+0000 7f1c78ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:56.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.968+0000 7f1c78ad9640 1 -- 172.21.15.102:0/1737699489 >> v1:172.21.15.102:6790/0 conn(0x7f1c74149ae0 legacy=0x7f1c7414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:56.974 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.968+0000 7f1c78ad9640 1 -- 172.21.15.102:0/1737699489 shutdown_connections 2024-03-25T18:36:56.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.968+0000 7f1c78ad9640 1 -- 172.21.15.102:0/1737699489 >> 172.21.15.102:0/1737699489 conn(0x7f1c740b5d10 msgr2=0x7f1c740b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:56.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.968+0000 7f1c78ad9640 1 -- 172.21.15.102:0/1737699489 shutdown_connections 2024-03-25T18:36:56.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.968+0000 7f1c78ad9640 1 -- 172.21.15.102:0/1737699489 wait complete. 2024-03-25T18:36:56.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 10 monclient: build_initial_monmap 2024-03-25T18:36:56.975 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 10 monclient: monmap: 2024-03-25T18:36:56.975 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:36:56.975 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:36:56.976 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:36:56.976 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:36:56.976 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:36:56.976 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:36:56.976 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:36:56.976 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:36:56.976 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:36:56.976 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 1 librados: starting msgr at 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 1 librados: starting objecter 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 1 Processor -- start 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 1 -- start start 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 1 librados: setting wanted keys 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 1 librados: calling monclient init 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.969+0000 7f1c78ad9640 10 monclient: init 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 10 monclient: _reopen_session rank -1 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 10 monclient(hunting): picked mon.noname-c con 0x7f1c740705b0 addr v1:172.21.15.102:6790/0 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 10 monclient(hunting): picked mon.noname-a con 0x7f1c741402d0 addr v1:172.21.15.102:6789/0 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 10 monclient(hunting): picked mon.noname-b con 0x7f1c74149ae0 addr v1:172.21.15.146:6789/0 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c74143b40 con 0x7f1c741402d0 2024-03-25T18:36:56.977 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c74144f50 con 0x7f1c74149ae0 2024-03-25T18:36:56.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c74145230 con 0x7f1c740705b0 2024-03-25T18:36:56.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 10 monclient(hunting): _renew_subs 2024-03-25T18:36:56.978 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.970+0000 7f1c78ad9640 10 monclient(hunting): authenticate will time out at 8048.258204s 2024-03-25T18:36:56.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.986+0000 7f1c71d9b640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1c741402d0 0x7f1c74134a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56914/0 (socket says 172.21.15.102:56914) 2024-03-25T18:36:56.987 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.986+0000 7f1c71d9b640 1 -- 172.21.15.102:0/3142931877 learned_addr learned my addr 172.21.15.102:0/3142931877 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:36:56.988 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.988+0000 7f1c62ffd640 10 client.?.objecter ms_handle_connect 0x7f1c741402d0 2024-03-25T18:36:56.988 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.988+0000 7f1c62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:56.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.990+0000 7f1c62ffd640 10 client.?.objecter ms_handle_connect 0x7f1c740705b0 2024-03-25T18:36:56.990 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.990+0000 7f1c62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:56.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.994+0000 7f1c62ffd640 10 client.?.objecter ms_handle_connect 0x7f1c74149ae0 2024-03-25T18:36:56.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.994+0000 7f1c62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:36:56.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686829499 0 0) 0x7f1c74145230 con 0x7f1c740705b0 2024-03-25T18:36:56.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:56.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:56.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:56.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:36:56.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c54004eb0 con 0x7f1c740705b0 2024-03-25T18:36:56.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995592094 0 0) 0x7f1c74143b40 con 0x7f1c741402d0 2024-03-25T18:36:56.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:56.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:56.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:56.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:36:56.999 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:56.997+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c74145230 con 0x7f1c741402d0 2024-03-25T18:36:57.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.009+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229441314 0 0) 0x7f1c74144f50 con 0x7f1c74149ae0 2024-03-25T18:36:57.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.009+0000 7f1c62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:36:57.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.009+0000 7f1c62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:36:57.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.009+0000 7f1c62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:36:57.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.009+0000 7f1c62ffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:36:57.010 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.009+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c74143b40 con 0x7f1c74149ae0 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.016+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3516527758 0 0) 0x7f1c74145230 con 0x7f1c741402d0 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.016+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c74144f50 con 0x7f1c741402d0 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.016+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1c68001a80 con 0x7f1c741402d0 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.016+0000 7f1c62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.016+0000 7f1c62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.016+0000 7f1c62ffd640 10 monclient(hunting): dump: 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:57.018 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.016+0000 7f1c62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.017+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 658489159 0 0) 0x7f1c54004eb0 con 0x7f1c740705b0 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.017+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c74145230 con 0x7f1c740705b0 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.017+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1c64001930 con 0x7f1c740705b0 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.017+0000 7f1c62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.017+0000 7f1c62ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.017+0000 7f1c62ffd640 10 monclient(hunting): dump: 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:57.019 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:57.020 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:57.020 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:57.020 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:57.020 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:57.020 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:57.020 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:57.020 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:57.020 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:57.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2292211021 0 0) 0x7f1c74144f50 con 0x7f1c741402d0 2024-03-25T18:36:57.029 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 >> v1:172.21.15.102:6790/0 conn(0x7f1c740705b0 legacy=0x7f1c74143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 >> v1:172.21.15.146:6789/0 conn(0x7f1c74149ae0 legacy=0x7f1c74135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 1 monclient: found mon.a 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c74145510 con 0x7f1c741402d0 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 10 monclient: _finish_auth 0 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c78ad9640 5 monclient: authenticate success, global_id 4716 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c78ad9640 10 monclient: _renew_subs 2024-03-25T18:36:57.030 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c78ad9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:57.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c74135880 con 0x7f1c741402d0 2024-03-25T18:36:57.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c78ad9640 10 client.4716.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:36:57.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c78ad9640 10 monclient: _renew_subs 2024-03-25T18:36:57.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c78ad9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:57.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.028+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7413faa0 con 0x7f1c741402d0 2024-03-25T18:36:57.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.029+0000 7f1c78ad9640 1 librados: init done 2024-03-25T18:36:57.031 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.030+0000 7f1c78ad9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:57.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.030+0000 7f1c78ad9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:57.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.030+0000 7f1c78ad9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:57.032 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.030+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c74143cf0 con 0x7f1c741402d0 2024-03-25T18:36:57.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.333+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c68001cc0 con 0x7f1c741402d0 2024-03-25T18:36:57.334 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.333+0000 7f1c62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.333+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f1c680038c0 con 0x7f1c741402d0 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.333+0000 7f1c62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.333+0000 7f1c62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.333+0000 7f1c62ffd640 10 monclient: dump: 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:36:57.335 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:36:57.336 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:36:57.336 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:36:57.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.333+0000 7f1c70d99640 4 set_mon_vals no callback set 2024-03-25T18:36:57.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.334+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f1c68003ae0 con 0x7f1c741402d0 2024-03-25T18:36:57.336 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.335+0000 7f1c7259c640 10 monclient: get_auth_request con 0x7f1c5406df60 auth_method 0 2024-03-25T18:36:57.337 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.335+0000 7f1c62ffd640 10 client.4716.objecter ms_handle_connect 0x7f1c5406df60 2024-03-25T18:36:57.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.339+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7f1c680546c0 con 0x7f1c741402d0 2024-03-25T18:36:57.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.339+0000 7f1c62ffd640 10 client.4716.objecter ms_dispatch 0x7f1c74000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:36:57.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.339+0000 7f1c62ffd640 3 client.4716.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:36:57.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.339+0000 7f1c62ffd640 3 client.4716.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:36:57.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.339+0000 7f1c62ffd640 20 client.4716.objecter dump_active .. 0 homeless 2024-03-25T18:36:57.340 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.339+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c680501d0 con 0x7f1c741402d0 2024-03-25T18:36:57.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.339+0000 7f1c62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:36:57.341 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.339+0000 7f1c62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:36:57.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.585+0000 7f1c78ad9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:57.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.585+0000 7f1c78ad9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:57.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.585+0000 7f1c78ad9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.102:6789/0 2024-03-25T18:36:57.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.585+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 --> v1:172.21.15.102:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c740627b0 con 0x7f1c741402d0 2024-03-25T18:36:57.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.618+0000 7f1c62ffd640 1 -- 172.21.15.102:0/3142931877 <== mon.0 v1:172.21.15.102:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7f1c68052100 con 0x7f1c741402d0 2024-03-25T18:36:57.618 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.618+0000 7f1c62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:36:57.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.618+0000 7f1c62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:36:57.619 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:36:57.619 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:36:57.621 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 10 librados: watch_flush enter 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 10 librados: watch_flush exit 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 20 client.4716.objecter shutdown clearing up homeless session... 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 10 client.4716.objecter successfully canceled tick 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 >> v1:172.21.15.146:6800/3848252714 conn(0x7f1c5406df60 legacy=0x7f1c54070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 10 monclient: shutdown 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 >> v1:172.21.15.102:6789/0 conn(0x7f1c741402d0 legacy=0x7f1c74134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 shutdown_connections 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 >> 172.21.15.102:0/3142931877 conn(0x7f1c740b5d10 msgr2=0x7f1c7413b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 shutdown_connections 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 1 -- 172.21.15.102:0/3142931877 wait complete. 2024-03-25T18:36:57.622 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:36:57.620+0000 7f1c78ad9640 1 librados: shutdown 2024-03-25T18:36:57.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:04.440 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:04.555 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.554+0000 7fc79d2d3640 10 monclient: get_monmap_and_config 2024-03-25T18:37:04.555 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.554+0000 7fc79d2d3640 10 monclient: build_initial_monmap 2024-03-25T18:37:04.555 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.554+0000 7fc79d2d3640 10 monclient: monmap: 2024-03-25T18:37:04.555 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:37:04.555 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:04.555 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:37:04.555386+0000 2024-03-25T18:37:04.556 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:37:04.555386+0000 2024-03-25T18:37:04.556 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:04.556 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:04.556 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:37:04.556 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:37:04.556 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:37:04.556 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:04.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.555+0000 7fc79d2d3640 1 Processor -- start 2024-03-25T18:37:04.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.555+0000 7fc79d2d3640 1 -- start start 2024-03-25T18:37:04.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.555+0000 7fc79d2d3640 10 monclient: init 2024-03-25T18:37:04.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:04.557 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:37:04.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 10 monclient(hunting): picked mon.noname-a con 0x7fc7980a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:37:04.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 10 monclient(hunting): picked mon.noname-c con 0x7fc79813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:37:04.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 10 monclient(hunting): picked mon.noname-b con 0x7fc7981430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:37:04.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc798142520 con 0x7fc7980a98f0 2024-03-25T18:37:04.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7980b28a0 con 0x7fc79813f710 2024-03-25T18:37:04.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7980b2a30 con 0x7fc7981430c0 2024-03-25T18:37:04.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:04.558 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.556+0000 7fc79d2d3640 10 monclient(hunting): authenticate will time out at 8055.844623s 2024-03-25T18:37:04.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.565+0000 7fc796ffd640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc7980a98f0 0x7fc7980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37226/0 (socket says 172.21.15.102:37226) 2024-03-25T18:37:04.565 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.565+0000 7fc796ffd640 1 -- 172.21.15.102:0/958691386 learned_addr learned my addr 172.21.15.102:0/958691386 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:37:04.576 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.576+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697160926 0 0) 0x7fc7980b2a30 con 0x7fc7981430c0 2024-03-25T18:37:04.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.576+0000 7fc795ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:37:04.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.576+0000 7fc795ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:04.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.576+0000 7fc795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:04.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.576+0000 7fc795ffb640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:37:04.577 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.576+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc780003520 con 0x7fc7981430c0 2024-03-25T18:37:04.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.577+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421459192 0 0) 0x7fc798142520 con 0x7fc7980a98f0 2024-03-25T18:37:04.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.577+0000 7fc795ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:37:04.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.577+0000 7fc795ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:04.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.577+0000 7fc795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:04.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.577+0000 7fc795ffb640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:37:04.578 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.577+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7980b2a30 con 0x7fc7980a98f0 2024-03-25T18:37:04.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.586+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302779155 0 0) 0x7fc7980b28a0 con 0x7fc79813f710 2024-03-25T18:37:04.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.586+0000 7fc795ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:37:04.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.586+0000 7fc795ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:04.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.586+0000 7fc795ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:04.586 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.586+0000 7fc795ffb640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:37:04.587 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.586+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc798142520 con 0x7fc79813f710 2024-03-25T18:37:04.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1265245923 0 0) 0x7fc780003520 con 0x7fc7981430c0 2024-03-25T18:37:04.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 >> v1:172.21.15.146:6789/0 conn(0x7fc79813f710 legacy=0x7fc7981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:04.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 >> v1:172.21.15.102:6789/0 conn(0x7fc7980a98f0 legacy=0x7fc7980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:04.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:37:04.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 1 monclient: found mon.noname-b 2024-03-25T18:37:04.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:04.590 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7980b44c0 con 0x7fc7981430c0 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 10 monclient: _finish_auth 0 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc79d2d3640 5 monclient: authenticate success, global_id 4703 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc790001a10 con 0x7fc7981430c0 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc795ffb640 10 monclient: dump: 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:04.591 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:04.592 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:04.592 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:04.592 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.589+0000 7fc79d2d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.597+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc790001c30 con 0x7fc7981430c0 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.597+0000 7fc795ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.597+0000 7fc795ffb640 1 -- 172.21.15.102:0/958691386 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc790001fc0 con 0x7fc7981430c0 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.597+0000 7fc795ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.597+0000 7fc795ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.597+0000 7fc795ffb640 10 monclient: dump: 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:04.598 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.597+0000 7fc79d2d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:04.599 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.597+0000 7fc79d2d3640 4 set_mon_vals no callback set 2024-03-25T18:37:04.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.598+0000 7fc79d2d3640 10 monclient: shutdown 2024-03-25T18:37:04.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.598+0000 7fc79d2d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:04.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.598+0000 7fc79d2d3640 1 -- 172.21.15.102:0/958691386 >> v1:172.21.15.102:6790/0 conn(0x7fc7981430c0 legacy=0x7fc79814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:04.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.598+0000 7fc79d2d3640 1 -- 172.21.15.102:0/958691386 shutdown_connections 2024-03-25T18:37:04.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.598+0000 7fc79d2d3640 1 -- 172.21.15.102:0/958691386 >> 172.21.15.102:0/958691386 conn(0x7fc7980b7ea0 msgr2=0x7fc7980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:04.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.598+0000 7fc79d2d3640 1 -- 172.21.15.102:0/958691386 shutdown_connections 2024-03-25T18:37:04.601 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.598+0000 7fc79d2d3640 1 -- 172.21.15.102:0/958691386 wait complete. 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.599+0000 7fc79d2d3640 10 monclient: build_initial_monmap 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.599+0000 7fc79d2d3640 10 monclient: monmap: 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.599+0000 7fc79d2d3640 1 librados: starting msgr at 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.599+0000 7fc79d2d3640 1 librados: starting objecter 2024-03-25T18:37:04.602 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.599+0000 7fc79d2d3640 1 Processor -- start 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 1 -- start start 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 1 librados: setting wanted keys 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 1 librados: calling monclient init 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 10 monclient: init 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 10 monclient(hunting): picked mon.noname-a con 0x7fc7980a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 10 monclient(hunting): picked mon.noname-b con 0x7fc79813f710 addr v1:172.21.15.146:6789/0 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 10 monclient(hunting): picked mon.noname-c con 0x7fc7981430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc798136ad0 con 0x7fc7980a98f0 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7980b7520 con 0x7fc79813f710 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc798135810 con 0x7fc7981430c0 2024-03-25T18:37:04.603 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:04.604 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.600+0000 7fc79d2d3640 10 monclient(hunting): authenticate will time out at 8055.888508s 2024-03-25T18:37:04.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.608+0000 7fc7977fe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7fc7981430c0 0x7fc798130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:41132/0 (socket says 172.21.15.102:41132) 2024-03-25T18:37:04.609 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.608+0000 7fc7977fe640 1 -- 172.21.15.102:0/199549177 learned_addr learned my addr 172.21.15.102:0/199549177 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:37:04.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.611+0000 7fc77f7fe640 10 client.?.objecter ms_handle_connect 0x7fc7981430c0 2024-03-25T18:37:04.611 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.611+0000 7fc77f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:04.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.614+0000 7fc77f7fe640 10 client.?.objecter ms_handle_connect 0x7fc7980a98f0 2024-03-25T18:37:04.615 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.614+0000 7fc77f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:04.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.619+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197318302 0 0) 0x7fc798135810 con 0x7fc7981430c0 2024-03-25T18:37:04.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.619+0000 7fc77f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:04.619 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.619+0000 7fc77f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:04.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.619+0000 7fc77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:04.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.619+0000 7fc77f7fe640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:37:04.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.619+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc784004eb0 con 0x7fc7981430c0 2024-03-25T18:37:04.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.620+0000 7fc77f7fe640 10 client.?.objecter ms_handle_connect 0x7fc79813f710 2024-03-25T18:37:04.620 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.620+0000 7fc77f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:04.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.627+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579771616 0 0) 0x7fc7980b7520 con 0x7fc79813f710 2024-03-25T18:37:04.628 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:04.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:04.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:04.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:37:04.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc798135810 con 0x7fc79813f710 2024-03-25T18:37:04.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402604400 0 0) 0x7fc798136ad0 con 0x7fc7980a98f0 2024-03-25T18:37:04.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:37:04.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:04.629 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:04.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:37:04.630 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.628+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7980b7520 con 0x7fc7980a98f0 2024-03-25T18:37:04.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.634+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 245974140 0 0) 0x7fc784004eb0 con 0x7fc7981430c0 2024-03-25T18:37:04.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.635+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc798136ad0 con 0x7fc7981430c0 2024-03-25T18:37:04.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.635+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc790001f40 con 0x7fc7981430c0 2024-03-25T18:37:04.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.635+0000 7fc77f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:04.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.635+0000 7fc77f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:04.635 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.635+0000 7fc77f7fe640 10 monclient(hunting): dump: 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:04.636 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.635+0000 7fc77f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:04.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.636+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 197278646 0 0) 0x7fc7980b7520 con 0x7fc7980a98f0 2024-03-25T18:37:04.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc784004eb0 con 0x7fc7980a98f0 2024-03-25T18:37:04.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc7880018f0 con 0x7fc7980a98f0 2024-03-25T18:37:04.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:04.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:04.637 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 10 monclient(hunting): dump: 2024-03-25T18:37:04.637 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:04.638 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 786991472 0 0) 0x7fc798135810 con 0x7fc79813f710 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7980b7520 con 0x7fc79813f710 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc78c001870 con 0x7fc79813f710 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.637+0000 7fc77f7fe640 10 monclient(hunting): dump: 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:04.639 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:04.640 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:04.640 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:04.640 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:04.643 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 63049017 0 0) 0x7fc7980b7520 con 0x7fc79813f710 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 >> v1:172.21.15.102:6790/0 conn(0x7fc7981430c0 legacy=0x7fc798130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 >> v1:172.21.15.102:6789/0 conn(0x7fc7980a98f0 legacy=0x7fc798135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 1 monclient: found mon.b 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7980b6180 con 0x7fc79813f710 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc77f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 10 monclient: _renew_subs 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7980b7a40 con 0x7fc79813f710 2024-03-25T18:37:04.644 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:04.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 10 monclient: _renew_subs 2024-03-25T18:37:04.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:04.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc798131840 con 0x7fc79813f710 2024-03-25T18:37:04.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.643+0000 7fc79d2d3640 1 librados: init done 2024-03-25T18:37:04.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.644+0000 7fc79d2d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:04.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.644+0000 7fc79d2d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:04.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.644+0000 7fc79d2d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:04.645 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.644+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc798136f60 con 0x7fc79813f710 2024-03-25T18:37:04.923 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.922+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc78c0035f0 con 0x7fc79813f710 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.923+0000 7fc77f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.923+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fc78c001d80 con 0x7fc79813f710 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.923+0000 7fc77f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.923+0000 7fc77f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.923+0000 7fc77f7fe640 10 monclient: dump: 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:04.924 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.923+0000 7fc7957fa640 4 set_mon_vals no callback set 2024-03-25T18:37:04.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.923+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fc78c019650 con 0x7fc79813f710 2024-03-25T18:37:04.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.924+0000 7fc796ffd640 10 monclient: get_auth_request con 0x7fc78406e160 auth_method 0 2024-03-25T18:37:04.925 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.924+0000 7fc77f7fe640 10 client.4735.objecter ms_handle_connect 0x7fc78406e160 2024-03-25T18:37:04.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.929+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7fc78c053380 con 0x7fc79813f710 2024-03-25T18:37:04.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.929+0000 7fc77f7fe640 10 client.4735.objecter ms_dispatch 0x7fc798000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:04.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.929+0000 7fc77f7fe640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:04.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.929+0000 7fc77f7fe640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:04.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.929+0000 7fc77f7fe640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T18:37:04.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.929+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc78c04ecc0 con 0x7fc79813f710 2024-03-25T18:37:04.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.929+0000 7fc77f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:04.930 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:04.929+0000 7fc77f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:05.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.173+0000 7fc79d2d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:05.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.173+0000 7fc79d2d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:05.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.173+0000 7fc79d2d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:05.174 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.173+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7980b6310 con 0x7fc79813f710 2024-03-25T18:37:05.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.201+0000 7fc77f7fe640 1 -- 172.21.15.102:0/199549177 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7fc78c050dc0 con 0x7fc79813f710 2024-03-25T18:37:05.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.201+0000 7fc77f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:05.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.201+0000 7fc77f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:05.202 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:37:05.202 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:05.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 10 librados: watch_flush enter 2024-03-25T18:37:05.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 10 librados: watch_flush exit 2024-03-25T18:37:05.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T18:37:05.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 10 client.4735.objecter successfully canceled tick 2024-03-25T18:37:05.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 >> v1:172.21.15.146:6800/3848252714 conn(0x7fc78406e160 legacy=0x7fc784070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:05.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 10 monclient: shutdown 2024-03-25T18:37:05.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:05.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 >> v1:172.21.15.146:6789/0 conn(0x7fc79813f710 legacy=0x7fc7980b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:05.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 shutdown_connections 2024-03-25T18:37:05.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 >> 172.21.15.102:0/199549177 conn(0x7fc7980b7ea0 msgr2=0x7fc798074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:05.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 shutdown_connections 2024-03-25T18:37:05.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 1 -- 172.21.15.102:0/199549177 wait complete. 2024-03-25T18:37:05.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:05.204+0000 7fc79d2d3640 1 librados: shutdown 2024-03-25T18:37:05.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:06.638 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:22.873 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:06.649 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:22.885 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:06.680 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:22.893 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:06.682 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:22.896 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:06.684 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:22.920 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:06.690 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:22.926 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:06.693 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:22.929 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:06.704 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:22.918 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:06.712 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:22.948 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.262 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:26.497 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.314 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:26.550 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.345 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:26.581 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.365 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:26.601 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.375 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:26.611 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.375 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:26.589 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.384 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:26.597 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.403 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:26.639 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:10.405 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:26.618 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:12.025 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:12.138 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.137+0000 7fa807e7b640 10 monclient: get_monmap_and_config 2024-03-25T18:37:12.138 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.137+0000 7fa807e7b640 10 monclient: build_initial_monmap 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.137+0000 7fa807e7b640 10 monclient: monmap: 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:37:12.138600+0000 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:37:12.138600+0000 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:37:12.139 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:12.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.138+0000 7fa807e7b640 1 Processor -- start 2024-03-25T18:37:12.140 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 1 -- start start 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 10 monclient: init 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa8000a98f0 addr v1:172.21.15.146:6789/0 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa80013f710 addr v1:172.21.15.102:6789/0 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8001430c0 addr v1:172.21.15.102:6790/0 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa800142520 con 0x7fa80013f710 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8000b28a0 con 0x7fa8000a98f0 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8000b2a30 con 0x7fa8001430c0 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:12.141 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.139+0000 7fa807e7b640 10 monclient(hunting): authenticate will time out at 8063.427714s 2024-03-25T18:37:12.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.156+0000 7fa805416640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa80013f710 0x7fa8001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55086/0 (socket says 172.21.15.102:55086) 2024-03-25T18:37:12.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.156+0000 7fa805c17640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa8000a98f0 0x7fa8000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.102:34778/0 (socket says 172.21.15.102:34778) 2024-03-25T18:37:12.156 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.156+0000 7fa805416640 1 -- 172.21.15.102:0/3241999132 learned_addr learned my addr 172.21.15.102:0/3241999132 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:37:12.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.168+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809546294 0 0) 0x7fa800142520 con 0x7fa80013f710 2024-03-25T18:37:12.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.168+0000 7fa804c15640 10 monclient(hunting): _negotiate 2024-03-25T18:37:12.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.168+0000 7fa804c15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:12.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.168+0000 7fa804c15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:12.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.168+0000 7fa804c15640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:37:12.169 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.168+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ec003520 con 0x7fa80013f710 2024-03-25T18:37:12.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.172+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374846382 0 0) 0x7fa8000b28a0 con 0x7fa8000a98f0 2024-03-25T18:37:12.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.172+0000 7fa804c15640 10 monclient(hunting): _negotiate 2024-03-25T18:37:12.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.172+0000 7fa804c15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:12.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.172+0000 7fa804c15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:12.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.172+0000 7fa804c15640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:37:12.173 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.172+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa800142520 con 0x7fa8000a98f0 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652863630 0 0) 0x7fa7ec003520 con 0x7fa80013f710 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 >> v1:172.21.15.102:6790/0 conn(0x7fa8001430c0 legacy=0x7fa80014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 >> v1:172.21.15.146:6789/0 conn(0x7fa8000a98f0 legacy=0x7fa8000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 10 monclient: _finish_hunting 0 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 1 monclient: found mon.noname-a 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8000b44c0 con 0x7fa80013f710 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 10 monclient: _finish_auth 0 2024-03-25T18:37:12.181 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 10 monclient: _check_auth_tickets 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa7f0001690 con 0x7fa80013f710 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa804c15640 10 monclient: dump: 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:12.182 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa807e7b640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:37:12.183 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.180+0000 7fa807e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:12.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.200+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7f00018b0 con 0x7fa80013f710 2024-03-25T18:37:12.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa804c15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:12.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa804c15640 1 -- 172.21.15.102:0/3241999132 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa7f0001c40 con 0x7fa80013f710 2024-03-25T18:37:12.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa804c15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:12.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa804c15640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:12.201 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa804c15640 10 monclient: dump: 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa807e7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:12.202 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa807e7b640 4 set_mon_vals no callback set 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa807e7b640 10 monclient: shutdown 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa807e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.201+0000 7fa807e7b640 1 -- 172.21.15.102:0/3241999132 >> v1:172.21.15.102:6789/0 conn(0x7fa80013f710 legacy=0x7fa8001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.202+0000 7fa807e7b640 1 -- 172.21.15.102:0/3241999132 shutdown_connections 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.202+0000 7fa807e7b640 1 -- 172.21.15.102:0/3241999132 >> 172.21.15.102:0/3241999132 conn(0x7fa8000b7ea0 msgr2=0x7fa8000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.202+0000 7fa807e7b640 1 -- 172.21.15.102:0/3241999132 shutdown_connections 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.202+0000 7fa807e7b640 1 -- 172.21.15.102:0/3241999132 wait complete. 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 10 monclient: build_initial_monmap 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 10 monclient: monmap: 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:12.204 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 1 librados: starting msgr at 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 1 librados: starting objecter 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 1 Processor -- start 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 1 -- start start 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 1 librados: setting wanted keys 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 1 librados: calling monclient init 2024-03-25T18:37:12.205 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.203+0000 7fa807e7b640 10 monclient: init 2024-03-25T18:37:12.206 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa8000a98f0 addr v1:172.21.15.102:6789/0 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa80013f710 addr v1:172.21.15.102:6790/0 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8001430c0 addr v1:172.21.15.146:6789/0 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8001370e0 con 0x7fa8000a98f0 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8000b7260 con 0x7fa8001430c0 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8000b73f0 con 0x7fa80013f710 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:12.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.204+0000 7fa807e7b640 10 monclient(hunting): authenticate will time out at 8063.492874s 2024-03-25T18:37:12.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.215+0000 7fa805c17640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa8000a98f0 0x7fa800137780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55130/0 (socket says 172.21.15.102:55130) 2024-03-25T18:37:12.216 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.215+0000 7fa805c17640 1 -- 172.21.15.102:0/1435499981 learned_addr learned my addr 172.21.15.102:0/1435499981 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:37:12.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.218+0000 7fa7f67fc640 10 client.?.objecter ms_handle_connect 0x7fa8001430c0 2024-03-25T18:37:12.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.218+0000 7fa7f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:12.218 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.218+0000 7fa7f67fc640 10 client.?.objecter ms_handle_connect 0x7fa8000a98f0 2024-03-25T18:37:12.219 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.218+0000 7fa7f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:12.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.221+0000 7fa7f67fc640 10 client.?.objecter ms_handle_connect 0x7fa80013f710 2024-03-25T18:37:12.222 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.221+0000 7fa7f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:12.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.229+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426626552 0 0) 0x7fa800142520 con 0x7fa8000a98f0 2024-03-25T18:37:12.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.229+0000 7fa7f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:12.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.229+0000 7fa7f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:12.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.229+0000 7fa7f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:12.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.229+0000 7fa7f67fc640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:37:12.230 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.229+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e4004eb0 con 0x7fa8000a98f0 2024-03-25T18:37:12.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.232+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795735269 0 0) 0x7fa8000b7260 con 0x7fa8001430c0 2024-03-25T18:37:12.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.232+0000 7fa7f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:12.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.232+0000 7fa7f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:12.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.232+0000 7fa7f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:12.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.232+0000 7fa7f67fc640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:37:12.232 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.232+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa800142520 con 0x7fa8001430c0 2024-03-25T18:37:12.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.243+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915096122 0 0) 0x7fa8000b73f0 con 0x7fa80013f710 2024-03-25T18:37:12.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.243+0000 7fa7f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:12.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.243+0000 7fa7f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:12.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.243+0000 7fa7f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:12.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.243+0000 7fa7f67fc640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:37:12.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.243+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8000b7260 con 0x7fa80013f710 2024-03-25T18:37:12.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.249+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 145082929 0 0) 0x7fa800142520 con 0x7fa8001430c0 2024-03-25T18:37:12.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.249+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8000b73f0 con 0x7fa8001430c0 2024-03-25T18:37:12.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.249+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa7f8001b00 con 0x7fa8001430c0 2024-03-25T18:37:12.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.249+0000 7fa7f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:12.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.249+0000 7fa7f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:12.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.249+0000 7fa7f67fc640 10 monclient(hunting): dump: 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:12.251 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.249+0000 7fa7f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.258+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2523765852 0 0) 0x7fa7e4004eb0 con 0x7fa8000a98f0 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.258+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa800142520 con 0x7fa8000a98f0 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.258+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa7fc001810 con 0x7fa8000a98f0 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.258+0000 7fa7f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.258+0000 7fa7f67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.258+0000 7fa7f67fc640 10 monclient(hunting): dump: 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:12.259 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:12.260 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:12.260 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:12.260 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:12.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4006773237 0 0) 0x7fa8000b73f0 con 0x7fa8001430c0 2024-03-25T18:37:12.263 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 >> v1:172.21.15.102:6790/0 conn(0x7fa80013f710 legacy=0x7fa8001daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 >> v1:172.21.15.102:6789/0 conn(0x7fa8000a98f0 legacy=0x7fa800137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 1 monclient: found mon.b 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa800139630 con 0x7fa8001430c0 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa7f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 5 monclient: authenticate success, global_id 4741 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 10 monclient: _renew_subs 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa800139880 con 0x7fa8001430c0 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:12.264 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 10 monclient: _renew_subs 2024-03-25T18:37:12.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:12.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8001332e0 con 0x7fa8001430c0 2024-03-25T18:37:12.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.262+0000 7fa807e7b640 1 librados: init done 2024-03-25T18:37:12.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.263+0000 7fa807e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:12.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.263+0000 7fa807e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:12.265 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.263+0000 7fa807e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:12.266 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.263+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa800136ee0 con 0x7fa8001430c0 2024-03-25T18:37:12.541 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.540+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7f8001fb0 con 0x7fa8001430c0 2024-03-25T18:37:12.541 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.540+0000 7fa7f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:12.541 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.540+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7fa7f8003b10 con 0x7fa8001430c0 2024-03-25T18:37:12.541 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.540+0000 7fa7f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:12.541 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.540+0000 7fa7f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:12.541 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.540+0000 7fa7f67fc640 10 monclient: dump: 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:12.542 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.540+0000 7fa7f7fff640 4 set_mon_vals no callback set 2024-03-25T18:37:12.543 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.541+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7fa7f8003d30 con 0x7fa8001430c0 2024-03-25T18:37:12.543 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.542+0000 7fa805c17640 10 monclient: get_auth_request con 0x7fa7e406df70 auth_method 0 2024-03-25T18:37:12.543 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.542+0000 7fa7f67fc640 10 client.4741.objecter ms_handle_connect 0x7fa7e406df70 2024-03-25T18:37:12.549 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.548+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7fa7f8053780 con 0x7fa8001430c0 2024-03-25T18:37:12.549 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.548+0000 7fa7f67fc640 10 client.4741.objecter ms_dispatch 0x7fa800000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:12.549 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.548+0000 7fa7f67fc640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:12.549 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.548+0000 7fa7f67fc640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:12.549 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.548+0000 7fa7f67fc640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T18:37:12.550 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.549+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7f804f290 con 0x7fa8001430c0 2024-03-25T18:37:12.550 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.549+0000 7fa7f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:12.550 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.549+0000 7fa7f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:12.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.769+0000 7fa807e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:12.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.769+0000 7fa807e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:12.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.769+0000 7fa807e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-03-25T18:37:12.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.769+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa800074cc0 con 0x7fa8001430c0 2024-03-25T18:37:12.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.832+0000 7fa7f67fc640 1 -- 172.21.15.102:0/1435499981 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7fa7f80511c0 con 0x7fa8001430c0 2024-03-25T18:37:12.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.832+0000 7fa7f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:12.833 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.832+0000 7fa7f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:12.833 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:37:12.834 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:12.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 10 librados: watch_flush enter 2024-03-25T18:37:12.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 10 librados: watch_flush exit 2024-03-25T18:37:12.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T18:37:12.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 10 client.4741.objecter successfully canceled tick 2024-03-25T18:37:12.836 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 >> v1:172.21.15.146:6800/3848252714 conn(0x7fa7e406df70 legacy=0x7fa7e4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:12.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 10 monclient: shutdown 2024-03-25T18:37:12.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:12.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 >> v1:172.21.15.146:6789/0 conn(0x7fa8001430c0 legacy=0x7fa8001367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:12.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 shutdown_connections 2024-03-25T18:37:12.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 >> 172.21.15.102:0/1435499981 conn(0x7fa8000b7ea0 msgr2=0x7fa80013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:12.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 shutdown_connections 2024-03-25T18:37:12.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 1 -- 172.21.15.102:0/1435499981 wait complete. 2024-03-25T18:37:12.837 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:12.835+0000 7fa807e7b640 1 librados: shutdown 2024-03-25T18:37:12.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:19.653 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:19.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.766+0000 7f2865b97640 10 monclient: get_monmap_and_config 2024-03-25T18:37:19.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.766+0000 7f2865b97640 10 monclient: build_initial_monmap 2024-03-25T18:37:19.768 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.766+0000 7f2865b97640 10 monclient: monmap: 2024-03-25T18:37:19.768 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:37:19.768 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:19.768 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:37:19.768541+0000 2024-03-25T18:37:19.768 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:37:19.768541+0000 2024-03-25T18:37:19.769 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:19.769 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:19.769 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:37:19.769 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:37:19.769 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:37:19.769 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:19.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.767+0000 7f2865b97640 1 Processor -- start 2024-03-25T18:37:19.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.767+0000 7f2865b97640 1 -- start start 2024-03-25T18:37:19.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.767+0000 7f2865b97640 10 monclient: init 2024-03-25T18:37:19.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:19.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:19.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 10 monclient(hunting): picked mon.noname-b con 0x7f286006c380 addr v1:172.21.15.102:6790/0 2024-03-25T18:37:19.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 10 monclient(hunting): picked mon.noname-a con 0x7f28600b12d0 addr v1:172.21.15.102:6789/0 2024-03-25T18:37:19.770 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 10 monclient(hunting): picked mon.noname-c con 0x7f28600aedf0 addr v1:172.21.15.146:6789/0 2024-03-25T18:37:19.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600b16f0 con 0x7f28600b12d0 2024-03-25T18:37:19.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600adfe0 con 0x7f28600aedf0 2024-03-25T18:37:19.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600add00 con 0x7f286006c380 2024-03-25T18:37:19.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:19.771 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.768+0000 7f2865b97640 10 monclient(hunting): authenticate will time out at 8071.057500s 2024-03-25T18:37:19.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.774+0000 7f285f7fe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f286006c380 0x7f28600b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:36296/0 (socket says 172.21.15.102:36296) 2024-03-25T18:37:19.775 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.774+0000 7f285f7fe640 1 -- 172.21.15.102:0/12791459 learned_addr learned my addr 172.21.15.102:0/12791459 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:37:19.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.785+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765998639 0 0) 0x7f28600add00 con 0x7f286006c380 2024-03-25T18:37:19.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.785+0000 7f285e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:19.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.785+0000 7f285e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:19.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.785+0000 7f285e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:19.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.785+0000 7f285e7fc640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:37:19.787 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.785+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2848003520 con 0x7f286006c380 2024-03-25T18:37:19.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.801+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563170339 0 0) 0x7f28600adfe0 con 0x7f28600aedf0 2024-03-25T18:37:19.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.801+0000 7f285e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:19.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.801+0000 7f285e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:19.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.801+0000 7f285e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:19.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.801+0000 7f285e7fc640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:37:19.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.801+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28600add00 con 0x7f28600aedf0 2024-03-25T18:37:19.812 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1281211037 0 0) 0x7f2848003520 con 0x7f286006c380 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 >> v1:172.21.15.146:6789/0 conn(0x7f28600aedf0 legacy=0x7f286014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 >> v1:172.21.15.102:6789/0 conn(0x7f28600b12d0 legacy=0x7f28600ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.102:6790/0 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600ae2c0 con 0x7f286006c380 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2850001690 con 0x7f286006c380 2024-03-25T18:37:19.813 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f2865b97640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f2865b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f285e7fc640 10 monclient: dump: 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:19.814 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.811+0000 7f2865b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:19.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.847+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 <== mon.2 v1:172.21.15.102:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28500017f0 con 0x7f286006c380 2024-03-25T18:37:19.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.847+0000 7f285e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:19.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.847+0000 7f285e7fc640 1 -- 172.21.15.102:0/12791459 <== mon.2 v1:172.21.15.102:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2850001b80 con 0x7f286006c380 2024-03-25T18:37:19.848 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.847+0000 7f285e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.847+0000 7f285e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.847+0000 7f285e7fc640 10 monclient: dump: 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.847+0000 7f2865b97640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:19.849 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.847+0000 7f2865b97640 4 set_mon_vals no callback set 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.848+0000 7f2865b97640 10 monclient: shutdown 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.848+0000 7f2865b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.848+0000 7f2865b97640 1 -- 172.21.15.102:0/12791459 >> v1:172.21.15.102:6790/0 conn(0x7f286006c380 legacy=0x7f28600b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.848+0000 7f2865b97640 1 -- 172.21.15.102:0/12791459 shutdown_connections 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.848+0000 7f2865b97640 1 -- 172.21.15.102:0/12791459 >> 172.21.15.102:0/12791459 conn(0x7f28600b5d10 msgr2=0x7f28600b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.848+0000 7f2865b97640 1 -- 172.21.15.102:0/12791459 shutdown_connections 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.848+0000 7f2865b97640 1 -- 172.21.15.102:0/12791459 wait complete. 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 10 monclient: build_initial_monmap 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 10 monclient: monmap: 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:37:19.851 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 1 librados: starting msgr at 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 1 librados: starting objecter 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 1 Processor -- start 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 1 -- start start 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 1 librados: setting wanted keys 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 1 librados: calling monclient init 2024-03-25T18:37:19.852 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.849+0000 7f2865b97640 10 monclient: init 2024-03-25T18:37:19.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:19.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:37:19.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 10 monclient(hunting): picked mon.noname-b con 0x7f286006c380 addr v1:172.21.15.146:6789/0 2024-03-25T18:37:19.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 10 monclient(hunting): picked mon.noname-a con 0x7f28600aedf0 addr v1:172.21.15.102:6789/0 2024-03-25T18:37:19.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 10 monclient(hunting): picked mon.noname-c con 0x7f28600b12d0 addr v1:172.21.15.102:6790/0 2024-03-25T18:37:19.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286006c8e0 con 0x7f28600aedf0 2024-03-25T18:37:19.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28601a39a0 con 0x7f286006c380 2024-03-25T18:37:19.853 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28601d7bf0 con 0x7f28600b12d0 2024-03-25T18:37:19.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:19.854 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.850+0000 7f2865b97640 10 monclient(hunting): authenticate will time out at 8071.139432s 2024-03-25T18:37:19.863 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.862+0000 7f285ffff640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f28600b12d0 0x7f28601c40f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:36350/0 (socket says 172.21.15.102:36350) 2024-03-25T18:37:19.864 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.862+0000 7f285ffff640 1 -- 172.21.15.102:0/2121346447 learned_addr learned my addr 172.21.15.102:0/2121346447 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:37:19.867 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.865+0000 7f2847fff640 10 client.?.objecter ms_handle_connect 0x7f286006c380 2024-03-25T18:37:19.867 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.865+0000 7f2847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:19.867 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.866+0000 7f2847fff640 10 client.?.objecter ms_handle_connect 0x7f28600b12d0 2024-03-25T18:37:19.867 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.866+0000 7f2847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:19.871 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.870+0000 7f2847fff640 10 client.?.objecter ms_handle_connect 0x7f28600aedf0 2024-03-25T18:37:19.872 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.870+0000 7f2847fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:19.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.874+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247193476 0 0) 0x7f28601d7bf0 con 0x7f28600b12d0 2024-03-25T18:37:19.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.874+0000 7f2847fff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:19.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.874+0000 7f2847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:19.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.874+0000 7f2847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:19.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.874+0000 7f2847fff640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:37:19.876 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.874+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284c004eb0 con 0x7f28600b12d0 2024-03-25T18:37:19.890 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.888+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033902702 0 0) 0x7f28601a39a0 con 0x7f286006c380 2024-03-25T18:37:19.890 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.888+0000 7f2847fff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:19.890 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.888+0000 7f2847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:19.890 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.888+0000 7f2847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:19.890 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.888+0000 7f2847fff640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:37:19.890 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.888+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28601d7bf0 con 0x7f286006c380 2024-03-25T18:37:19.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.889+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889227384 0 0) 0x7f286006c8e0 con 0x7f28600aedf0 2024-03-25T18:37:19.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.890+0000 7f2847fff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:19.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.890+0000 7f2847fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:19.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.890+0000 7f2847fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:19.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.890+0000 7f2847fff640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:37:19.891 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.890+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28601a39a0 con 0x7f28600aedf0 2024-03-25T18:37:19.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.900+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3836597623 0 0) 0x7f284c004eb0 con 0x7f28600b12d0 2024-03-25T18:37:19.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.900+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f286006c8e0 con 0x7f28600b12d0 2024-03-25T18:37:19.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.900+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2858001d10 con 0x7f28600b12d0 2024-03-25T18:37:19.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.900+0000 7f2847fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:19.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.900+0000 7f2847fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:19.901 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.900+0000 7f2847fff640 10 monclient(hunting): dump: 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:19.902 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.900+0000 7f2847fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.902+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1186087358 0 0) 0x7f28601d7bf0 con 0x7f286006c380 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.902+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f284c004eb0 con 0x7f286006c380 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.902+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2850001b00 con 0x7f286006c380 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.902+0000 7f2847fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.902+0000 7f2847fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.902+0000 7f2847fff640 10 monclient(hunting): dump: 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.904 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:19.905 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:19.905 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:19.905 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:19.905 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:19.905 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.915+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4141666487 0 0) 0x7f28601a39a0 con 0x7f28600aedf0 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.915+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28601d7bf0 con 0x7f28600aedf0 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.915+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f28540017c0 con 0x7f28600aedf0 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.915+0000 7f2847fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.915+0000 7f2847fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.915+0000 7f2847fff640 10 monclient(hunting): dump: 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:19.917 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.918 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.918 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:19.918 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:19.918 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:19.918 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:19.918 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:19.918 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:19.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.924+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3824634028 0 0) 0x7f286006c8e0 con 0x7f28600b12d0 2024-03-25T18:37:19.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 >> v1:172.21.15.146:6789/0 conn(0x7f286006c380 legacy=0x7f28601d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:19.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 >> v1:172.21.15.102:6789/0 conn(0x7f28600aedf0 legacy=0x7f28601c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:19.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 10 monclient: _finish_hunting 0 2024-03-25T18:37:19.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 1 monclient: found mon.c 2024-03-25T18:37:19.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:19.927 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28601c4800 con 0x7f28600b12d0 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 10 monclient: _finish_auth 0 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 10 monclient: _check_auth_tickets 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 10 monclient: _renew_subs 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28601a3480 con 0x7f28600b12d0 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 10 client.4718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 10 monclient: _renew_subs 2024-03-25T18:37:19.928 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:19.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28601c51e0 con 0x7f28600b12d0 2024-03-25T18:37:19.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.925+0000 7f2865b97640 1 librados: init done 2024-03-25T18:37:19.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.926+0000 7f2865b97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:19.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.926+0000 7f2865b97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:19.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.926+0000 7f2865b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:19.929 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.926+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286006d770 con 0x7f28600b12d0 2024-03-25T18:37:19.953 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.952+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2858003990 con 0x7f28600b12d0 2024-03-25T18:37:19.953 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.952+0000 7f2847fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:19.953 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.952+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f2858003d60 con 0x7f28600b12d0 2024-03-25T18:37:19.953 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.952+0000 7f2847fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.952+0000 7f2847fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.952+0000 7f2847fff640 10 monclient: dump: 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:19.954 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.952+0000 7f285dffb640 4 set_mon_vals no callback set 2024-03-25T18:37:19.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.953+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f2858003f80 con 0x7f28600b12d0 2024-03-25T18:37:19.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.953+0000 7f285f7fe640 10 monclient: get_auth_request con 0x7f284c06e060 auth_method 0 2024-03-25T18:37:19.955 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.953+0000 7f2847fff640 10 client.4718.objecter ms_handle_connect 0x7f284c06e060 2024-03-25T18:37:19.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.956+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7f2858054b80 con 0x7f28600b12d0 2024-03-25T18:37:19.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.956+0000 7f2847fff640 10 client.4718.objecter ms_dispatch 0x7f2860000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:19.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.956+0000 7f2847fff640 3 client.4718.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:19.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.956+0000 7f2847fff640 3 client.4718.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:19.958 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:19.956+0000 7f2847fff640 20 client.4718.objecter dump_active .. 0 homeless 2024-03-25T18:37:20.243 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.242+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2858050690 con 0x7f28600b12d0 2024-03-25T18:37:20.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.242+0000 7f2847fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:20.244 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.242+0000 7f2847fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:20.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.466+0000 7f2865b97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:20.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.466+0000 7f2865b97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:20.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.466+0000 7f2865b97640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:20.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.466+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28601a37d0 con 0x7f28600b12d0 2024-03-25T18:37:20.494 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.492+0000 7f2847fff640 1 -- 172.21.15.102:0/2121346447 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7f28580525c0 con 0x7f28600b12d0 2024-03-25T18:37:20.494 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.493+0000 7f2847fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:20.494 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.493+0000 7f2847fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:20.495 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:37:20.495 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:20.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 10 librados: watch_flush enter 2024-03-25T18:37:20.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 10 librados: watch_flush exit 2024-03-25T18:37:20.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 20 client.4718.objecter shutdown clearing up homeless session... 2024-03-25T18:37:20.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 10 client.4718.objecter successfully canceled tick 2024-03-25T18:37:20.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 >> v1:172.21.15.146:6800/3848252714 conn(0x7f284c06e060 legacy=0x7f284c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:20.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 10 monclient: shutdown 2024-03-25T18:37:20.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:20.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 >> v1:172.21.15.102:6790/0 conn(0x7f28600b12d0 legacy=0x7f28601c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:20.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 shutdown_connections 2024-03-25T18:37:20.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 >> 172.21.15.102:0/2121346447 conn(0x7f28600b5d10 msgr2=0x7f28600b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:20.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 shutdown_connections 2024-03-25T18:37:20.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.495+0000 7f2865b97640 1 -- 172.21.15.102:0/2121346447 wait complete. 2024-03-25T18:37:20.498 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:20.496+0000 7f2865b97640 1 librados: shutdown 2024-03-25T18:37:20.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:27.314 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.425+0000 7f33fe125640 10 monclient: get_monmap_and_config 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.425+0000 7f33fe125640 10 monclient: build_initial_monmap 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.425+0000 7f33fe125640 10 monclient: monmap: 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:37:27.427219+0000 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:37:27.427219+0000 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-03-25T18:37:27.427 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-b 2024-03-25T18:37:27.428 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.426+0000 7f33fe125640 1 Processor -- start 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.426+0000 7f33fe125640 1 -- start start 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.426+0000 7f33fe125640 10 monclient: init 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 10 monclient(hunting): picked mon.noname-a con 0x7f33f80b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 10 monclient(hunting): picked mon.noname-b con 0x7f33f80b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 10 monclient(hunting): picked mon.noname-c con 0x7f33f814be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f80af940 con 0x7f33f80b4d60 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f80b0a70 con 0x7f33f814be90 2024-03-25T18:37:27.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f80b0d50 con 0x7f33f80b1730 2024-03-25T18:37:27.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:27.430 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.427+0000 7f33fe125640 10 monclient(hunting): authenticate will time out at 8078.716319s 2024-03-25T18:37:27.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.435+0000 7f33f77fe640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f33f80b4d60 0x7f33f813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42530/0 (socket says 172.21.15.102:42530) 2024-03-25T18:37:27.436 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.435+0000 7f33f77fe640 1 -- 172.21.15.102:0/3934171782 learned_addr learned my addr 172.21.15.102:0/3934171782 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:37:27.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.442+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827317901 0 0) 0x7f33f80af940 con 0x7f33f80b4d60 2024-03-25T18:37:27.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.442+0000 7f33f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:27.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.442+0000 7f33f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:27.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.442+0000 7f33f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:27.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.442+0000 7f33f67fc640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:37:27.445 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.443+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e0003520 con 0x7f33f80b4d60 2024-03-25T18:37:27.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.454+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250477166 0 0) 0x7f33f80b0a70 con 0x7f33f814be90 2024-03-25T18:37:27.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.454+0000 7f33f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:37:27.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.454+0000 7f33f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:27.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.454+0000 7f33f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:27.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.454+0000 7f33f67fc640 10 monclient(hunting): my global_id is 4750 2024-03-25T18:37:27.456 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.454+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f80af940 con 0x7f33f814be90 2024-03-25T18:37:27.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4293105557 0 0) 0x7f33e0003520 con 0x7f33f80b4d60 2024-03-25T18:37:27.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 >> v1:172.21.15.102:6790/0 conn(0x7f33f80b1730 legacy=0x7f33f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:27.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 >> v1:172.21.15.146:6789/0 conn(0x7f33f814be90 legacy=0x7f33f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:27.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:37:27.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 1 monclient: found mon.noname-a 2024-03-25T18:37:27.458 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.102:6789/0 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f80b1030 con 0x7f33f80b4d60 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 10 monclient: _finish_auth 0 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f33e8001690 con 0x7f33f80b4d60 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33fe125640 5 monclient: authenticate success, global_id 4737 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33fe125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33f67fc640 10 monclient: dump: 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:27.459 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:27.460 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:27.460 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:27.460 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:27.460 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:27.460 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:27.460 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:27.460 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:27.460 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.456+0000 7f33fe125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:37:27.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.461+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 <== mon.0 v1:172.21.15.102:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33e80017f0 con 0x7f33f80b4d60 2024-03-25T18:37:27.462 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.461+0000 7f33f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.461+0000 7f33f67fc640 1 -- 172.21.15.102:0/3934171782 <== mon.0 v1:172.21.15.102:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f33e8001c00 con 0x7f33f80b4d60 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.461+0000 7f33fe125640 10 monclient: get_monmap_and_config success 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.461+0000 7f33fe125640 4 set_mon_vals no callback set 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.461+0000 7f33f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.461+0000 7f33f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.461+0000 7f33f67fc640 10 monclient: dump: 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:27.463 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:27.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.462+0000 7f33fe125640 10 monclient: shutdown 2024-03-25T18:37:27.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.462+0000 7f33fe125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:27.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.462+0000 7f33fe125640 1 -- 172.21.15.102:0/3934171782 >> v1:172.21.15.102:6789/0 conn(0x7f33f80b4d60 legacy=0x7f33f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:27.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.462+0000 7f33fe125640 1 -- 172.21.15.102:0/3934171782 shutdown_connections 2024-03-25T18:37:27.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.462+0000 7f33fe125640 1 -- 172.21.15.102:0/3934171782 >> 172.21.15.102:0/3934171782 conn(0x7f33f80617a0 msgr2=0x7f33f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:27.465 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.462+0000 7f33fe125640 1 -- 172.21.15.102:0/3934171782 shutdown_connections 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.462+0000 7f33fe125640 1 -- 172.21.15.102:0/3934171782 wait complete. 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.463+0000 7f33fe125640 10 monclient: build_initial_monmap 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.463+0000 7f33fe125640 10 monclient: monmap: 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:epoch 0 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:last_changed 0.000000 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:created 0.000000 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 0 (unknown) 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 1 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.noname-a 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.noname-c 2024-03-25T18:37:27.466 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:27.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.463+0000 7f33fe125640 1 librados: starting msgr at 2024-03-25T18:37:27.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.463+0000 7f33fe125640 1 librados: starting objecter 2024-03-25T18:37:27.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.463+0000 7f33fe125640 1 Processor -- start 2024-03-25T18:37:27.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 1 -- start start 2024-03-25T18:37:27.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 1 librados: setting wanted keys 2024-03-25T18:37:27.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 1 librados: calling monclient init 2024-03-25T18:37:27.467 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 10 monclient: init 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 10 monclient: _reopen_session rank -1 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 10 monclient(hunting): picked mon.noname-c con 0x7f33f80b1730 addr v1:172.21.15.102:6790/0 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 10 monclient(hunting): picked mon.noname-a con 0x7f33f80b4d60 addr v1:172.21.15.102:6789/0 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 10 monclient(hunting): picked mon.noname-b con 0x7f33f814be90 addr v1:172.21.15.146:6789/0 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f81368f0 con 0x7f33f80b4d60 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f8137750 con 0x7f33f814be90 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 1 -- --> v1:172.21.15.102:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f81363d0 con 0x7f33f80b1730 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 10 monclient(hunting): _renew_subs 2024-03-25T18:37:27.468 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.464+0000 7f33fe125640 10 monclient(hunting): authenticate will time out at 8078.753772s 2024-03-25T18:37:27.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.475+0000 7f33f77fe640 1 --1- >> v1:172.21.15.102:6790/0 conn(0x7f33f80b1730 0x7f33f8135b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6790/0 says I am v1:172.21.15.102:34742/0 (socket says 172.21.15.102:34742) 2024-03-25T18:37:27.476 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.475+0000 7f33f77fe640 1 -- 172.21.15.102:0/1903307749 learned_addr learned my addr 172.21.15.102:0/1903307749 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-25T18:37:27.477 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.476+0000 7f33dffff640 10 client.?.objecter ms_handle_connect 0x7f33f80b1730 2024-03-25T18:37:27.477 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.476+0000 7f33dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:27.483 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.481+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689974232 0 0) 0x7f33f81363d0 con 0x7f33f80b1730 2024-03-25T18:37:27.483 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.481+0000 7f33dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:27.483 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.481+0000 7f33dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:27.483 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.481+0000 7f33dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:27.483 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.481+0000 7f33dffff640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:37:27.483 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.481+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.102:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e4004eb0 con 0x7f33f80b1730 2024-03-25T18:37:27.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.483+0000 7f33dffff640 10 client.?.objecter ms_handle_connect 0x7f33f80b4d60 2024-03-25T18:37:27.484 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.483+0000 7f33dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:27.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.488+0000 7f33dffff640 10 client.?.objecter ms_handle_connect 0x7f33f814be90 2024-03-25T18:37:27.489 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.488+0000 7f33dffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:37:27.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.495+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4082326992 0 0) 0x7f33e4004eb0 con 0x7f33f80b1730 2024-03-25T18:37:27.497 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.495+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.102:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33f81363d0 con 0x7f33f80b1730 2024-03-25T18:37:27.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.497+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762553547 0 0) 0x7f33f8137750 con 0x7f33f814be90 2024-03-25T18:37:27.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.497+0000 7f33dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:27.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.497+0000 7f33dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:27.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.497+0000 7f33dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:27.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.497+0000 7f33dffff640 10 monclient(hunting): my global_id is 4753 2024-03-25T18:37:27.499 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.497+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e4004eb0 con 0x7f33f814be90 2024-03-25T18:37:27.502 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707711359 0 0) 0x7f33f81368f0 con 0x7f33f80b4d60 2024-03-25T18:37:27.502 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f8137750 con 0x7f33f80b4d60 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f33e8001ca0 con 0x7f33f80b1730 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 10 monclient(hunting): dump: 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:27.503 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:27.504 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:27.504 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:27.504 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:27.504 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:27.504 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.501+0000 7f33dffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:37:27.507 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4077816192 0 0) 0x7f33f81363d0 con 0x7f33f80b1730 2024-03-25T18:37:27.507 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 >> v1:172.21.15.146:6789/0 conn(0x7f33f814be90 legacy=0x7f33f81d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 >> v1:172.21.15.102:6789/0 conn(0x7f33f80b4d60 legacy=0x7f33f812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 10 monclient: _finish_hunting 0 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 1 monclient: found mon.c 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.102:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f8130470 con 0x7f33f80b1730 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 10 monclient: _finish_auth 0 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.505+0000 7f33dffff640 10 monclient: _check_auth_tickets 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 5 monclient: authenticate success, global_id 4724 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 10 monclient: _renew_subs 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.102:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f8130720 con 0x7f33f80b1730 2024-03-25T18:37:27.508 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:37:27.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 10 monclient: _renew_subs 2024-03-25T18:37:27.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:27.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.102:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f81322f0 con 0x7f33f80b1730 2024-03-25T18:37:27.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 1 librados: init done 2024-03-25T18:37:27.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:27.509 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:27.510 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:27.510 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.506+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f8136210 con 0x7f33f80b1730 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.800+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33e80011f0 con 0x7f33f80b1730 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.800+0000 7f33dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.800+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2963267054 0 0) 0x7f33e8004cb0 con 0x7f33f80b1730 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.800+0000 7f33dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.800+0000 7f33dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.800+0000 7f33dffff640 10 monclient: dump: 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:epoch 1 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:fsid 56a4cafe-eb50-4500-9868-7090e2ea20fc 2024-03-25T18:37:27.802 INFO:teuthology.misc.health.smithi102.stderr:last_changed 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:27.803 INFO:teuthology.misc.health.smithi102.stderr:created 2024-03-25T18:25:10.892074+0000 2024-03-25T18:37:27.803 INFO:teuthology.misc.health.smithi102.stderr:min_mon_release 19 (squid) 2024-03-25T18:37:27.803 INFO:teuthology.misc.health.smithi102.stderr:election_strategy: 3 2024-03-25T18:37:27.803 INFO:teuthology.misc.health.smithi102.stderr:0: v1:172.21.15.102:6789/0 mon.a 2024-03-25T18:37:27.803 INFO:teuthology.misc.health.smithi102.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-03-25T18:37:27.803 INFO:teuthology.misc.health.smithi102.stderr:2: v1:172.21.15.102:6790/0 mon.c 2024-03-25T18:37:27.803 INFO:teuthology.misc.health.smithi102.stderr: 2024-03-25T18:37:27.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.800+0000 7f33f5ffb640 4 set_mon_vals no callback set 2024-03-25T18:37:27.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.801+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3296226369 0 0) 0x7f33e801ad40 con 0x7f33f80b1730 2024-03-25T18:37:27.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.802+0000 7f33f6ffd640 10 monclient: get_auth_request con 0x7f33e406dda0 auth_method 0 2024-03-25T18:37:27.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.802+0000 7f33dffff640 10 client.4724.objecter ms_handle_connect 0x7f33e406dda0 2024-03-25T18:37:27.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.807+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3021377712 0 0) 0x7f33e8054a00 con 0x7f33f80b1730 2024-03-25T18:37:27.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.807+0000 7f33dffff640 10 client.4724.objecter ms_dispatch 0x7f33f8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:37:27.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.807+0000 7f33dffff640 3 client.4724.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:37:27.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.807+0000 7f33dffff640 3 client.4724.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:37:27.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.807+0000 7f33dffff640 20 client.4724.objecter dump_active .. 0 homeless 2024-03-25T18:37:27.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.807+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33e8050340 con 0x7f33f80b1730 2024-03-25T18:37:27.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.807+0000 7f33dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:37:27.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:27.807+0000 7f33dffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:37:28.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.061+0000 7f33fe125640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:28.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.061+0000 7f33fe125640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:28.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.061+0000 7f33fe125640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.102:6790/0 2024-03-25T18:37:28.063 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.061+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 --> v1:172.21.15.102:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33f80b2270 con 0x7f33f80b1730 2024-03-25T18:37:28.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.095+0000 7f33dffff640 1 -- 172.21.15.102:0/1903307749 <== mon.2 v1:172.21.15.102:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2280223005) 0x7f33e8052440 con 0x7f33f80b1730 2024-03-25T18:37:28.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.095+0000 7f33dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:37:28.097 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.095+0000 7f33dffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:37:28.097 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-25T18:37:28.098 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":11,"fsid":"56a4cafe-eb50-4500-9868-7090e2ea20fc","created":"2024-03-25T18:25:59.909172+0000","modified":"2024-03-25T18:36:04.966597+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:36:04.966597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d665d3c6-ac37-40ea-b391-93ce0dd4500b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb6f7c78-451d-4f40-b363-5de7e3f0fcb9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"97cab3fa-01da-417b-ba9e-12ad9c9792e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"ad32c648-b51d-4308-9317-5d70afde38f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53b511a1-22cc-445a-91fb-e4d7cbd8a021","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"584db0cd-8ea4-4a16-b14b-12197ed63ee4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f0686c05-1204-4260-b598-6ac099db8c16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"22087808-2aab-4158-ba49-c99624a3580c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:37:28.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.097+0000 7f33fe125640 10 librados: watch_flush enter 2024-03-25T18:37:28.100 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.097+0000 7f33fe125640 10 librados: watch_flush exit 2024-03-25T18:37:28.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.097+0000 7f33fe125640 20 client.4724.objecter shutdown clearing up homeless session... 2024-03-25T18:37:28.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.097+0000 7f33fe125640 10 client.4724.objecter successfully canceled tick 2024-03-25T18:37:28.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 >> v1:172.21.15.146:6800/3848252714 conn(0x7f33e406dda0 legacy=0x7f33e4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:28.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 10 monclient: shutdown 2024-03-25T18:37:28.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:37:28.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 >> v1:172.21.15.102:6790/0 conn(0x7f33f80b1730 legacy=0x7f33f8135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:37:28.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 shutdown_connections 2024-03-25T18:37:28.101 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 >> 172.21.15.102:0/1903307749 conn(0x7f33f80617a0 msgr2=0x7f33f8138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:37:28.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 shutdown_connections 2024-03-25T18:37:28.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 1 -- 172.21.15.102:0/1903307749 wait complete. 2024-03-25T18:37:28.102 INFO:teuthology.misc.health.smithi102.stderr:2024-03-25T18:37:28.098+0000 7f33fe125640 1 librados: shutdown 2024-03-25T18:37:28.111 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:37:28.111 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:37:28.113 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:37:28.113 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:37:28.114 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:30.158 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:11:23.840 35533== 2024-03-25T18:37:30.158 INFO:tasks.ceph.osd.0.smithi102.stderr:==00:00:11:23.840 35533== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:30.197 INFO:tasks.ceph.osd.0.smithi102.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:34.216 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:37:34.217 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:37:34.217 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:37:34.217 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:35.374 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:51.609 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.424 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:51.660 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.454 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:11:51.690 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.483 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:51.697 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.489 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:51.702 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.500 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:51.736 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.506 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:51.742 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.511 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:11:51.724 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:35.534 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:11:51.770 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:36.195 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:11:29.877 35532== 2024-03-25T18:37:36.195 INFO:tasks.ceph.osd.1.smithi102.stderr:==00:00:11:29.877 35532== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:36.235 INFO:tasks.ceph.osd.1.smithi102.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:40.319 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:37:40.319 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:37:40.320 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:37:40.320 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:42.325 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:11:36.007 35531== 2024-03-25T18:37:42.325 INFO:tasks.ceph.osd.2.smithi102.stderr:==00:00:11:36.007 35531== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:42.363 INFO:tasks.ceph.osd.2.smithi102.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:46.421 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:46.422 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:37:46.422 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:37:46.422 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:37:46.422 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:48.396 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:11:42.067 35536== 2024-03-25T18:37:48.396 INFO:tasks.ceph.osd.3.smithi102.stderr:==00:00:11:42.067 35536== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:48.438 INFO:tasks.ceph.osd.3.smithi102.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:52.524 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:37:52.524 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:37:52.525 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:37:52.525 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:37:54.637 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:11:48.303 35221== 2024-03-25T18:37:54.637 INFO:tasks.ceph.osd.4.smithi146.stderr:==00:00:11:48.303 35221== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:37:54.678 INFO:tasks.ceph.osd.4.smithi146.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:37:55.450 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:12:11.686 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:55.460 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:12:11.696 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:55.481 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:12:11.717 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:55.497 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:12:11.711 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:55.501 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:12:11.714 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:55.507 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:11.743 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:55.510 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:11.746 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:55.522 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:12:11.736 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:55.542 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:11.778 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:37:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:37:58.626 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:37:58.627 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:37:58.627 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:37:58.627 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:00.604 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:11:54.270 35222== 2024-03-25T18:38:00.604 INFO:tasks.ceph.osd.5.smithi146.stderr:==00:00:11:54.270 35222== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:00.642 INFO:tasks.ceph.osd.5.smithi146.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:04.729 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:38:04.729 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:38:04.730 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:38:04.730 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:06.581 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:12:00.238 35224== 2024-03-25T18:38:06.581 INFO:tasks.ceph.osd.6.smithi146.stderr:==00:00:12:00.238 35224== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:06.634 INFO:tasks.ceph.osd.6.smithi146.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:10.832 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:38:10.832 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:38:10.833 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:38:10.833 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:12.409 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:12:06.067 35223== 2024-03-25T18:38:12.410 INFO:tasks.ceph.osd.7.smithi146.stderr:==00:00:12:06.067 35223== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:12.443 INFO:tasks.ceph.osd.7.smithi146.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:15.503 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:12:31.739 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:15.507 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:12:31.743 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:15.518 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:12:31.754 35224== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:15.552 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:31.788 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:15.553 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:12:31.767 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:15.553 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:31.790 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:15.555 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:12:31.768 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:15.565 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:31.801 35220== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:15.566 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:12:31.780 34611== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:16.935 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:38:16.935 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:38:16.935 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:38:16.936 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:38:16.936 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:16.959 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:38:16.960 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:38:16.960 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:16.961 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:38:16.961+0000 7f51d7f51640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34609) UID: 0 2024-03-25T18:38:16.961 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-25T18:38:16.961+0000 7f51d7f51640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:38:16.988 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:38:16.988 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:38:16.988 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:38:16.988 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:17.014 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-25T18:38:17.002+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35189) UID: 0 2024-03-25T18:38:17.014 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-25T18:38:17.003+0000 9f86640 -1 mon.a@0(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:38:17.498 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:33.734 35220== 2024-03-25T18:38:17.499 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:33.734 35220== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:17.507 INFO:tasks.ceph.mon.a.smithi102.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:23.090 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:23.090 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:38:23.091 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:38:23.091 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:38:23.091 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:23.107 INFO:tasks.ceph.mon.c.smithi102.stderr:2024-03-25T18:38:23.094+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35191) UID: 0 2024-03-25T18:38:23.107 INFO:tasks.ceph.mon.c.smithi102.stderr:2024-03-25T18:38:23.097+0000 9f86640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:38:23.769 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:12:40.004 35224== 2024-03-25T18:38:23.769 INFO:tasks.ceph.mon.c.smithi102.stderr:==00:00:12:40.004 35224== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:23.778 INFO:tasks.ceph.mon.c.smithi102.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:29.193 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:38:29.193 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:38:29.193 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:38:29.193 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:38:29.214 INFO:tasks.ceph.mon.b.smithi146.stderr:2024-03-25T18:38:29.198+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34607) UID: 0 2024-03-25T18:38:29.214 INFO:tasks.ceph.mon.b.smithi146.stderr:2024-03-25T18:38:29.199+0000 9f86640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:38:29.632 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:12:45.845 34611== 2024-03-25T18:38:29.632 INFO:tasks.ceph.mon.b.smithi146.stderr:==00:00:12:45.845 34611== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:38:29.641 INFO:tasks.ceph.mon.b.smithi146.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:38:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:38:35.296 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:38:35.296 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:38:35.296 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:38:35.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:38:35.341 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:38:35.342 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:38:35.559 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:38:35.560 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:38:35.631 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:38:35.631 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:38:35.691 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:38:35.691 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:38:35.747 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:38:35.747 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:38:35.894 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:38:35.895 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:38:35.957 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:38:35.958 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:38:36.015 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:38:36.015 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:38:36.082 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:38:36.084 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/data/mon.a.tgz 2024-03-25T18:38:36.085 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-25T18:38:36.100 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.rpilgKjvf2 2024-03-25T18:38:36.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.rpilgKjvf2 2024-03-25T18:38:36.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.rpilgKjvf2 2024-03-25T18:38:36.288 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.rpilgKjvf2 is 132KB 2024-03-25T18:38:36.339 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.rpilgKjvf2 2024-03-25T18:38:36.353 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/data/mon.c.tgz 2024-03-25T18:38:36.354 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-25T18:38:36.407 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.dSlLnABaxb 2024-03-25T18:38:36.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.dSlLnABaxb 2024-03-25T18:38:36.494 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.dSlLnABaxb 2024-03-25T18:38:36.578 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.dSlLnABaxb is 132KB 2024-03-25T18:38:36.629 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.dSlLnABaxb 2024-03-25T18:38:36.643 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/data/mon.b.tgz 2024-03-25T18:38:36.643 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-25T18:38:36.659 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.VUtwhGo2FU 2024-03-25T18:38:36.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VUtwhGo2FU 2024-03-25T18:38:36.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.VUtwhGo2FU 2024-03-25T18:38:36.845 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.VUtwhGo2FU is 132KB 2024-03-25T18:38:36.896 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.VUtwhGo2FU 2024-03-25T18:38:36.911 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:38:36.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:38:36.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:38:37.269 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:38:37.538 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:38:37.540 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/remote/smithi102/crash 2024-03-25T18:38:37.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:38:37.571 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/remote/smithi146/crash 2024-03-25T18:38:37.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:38:37.598 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:38:37.598 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:38:37.613 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:38:37.690 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:38:37.690 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34814.log 2024-03-25T18:38:37.690 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34814.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34814.log.gz 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:38:37.691 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35028.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34856.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.35028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34765.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34856.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35028.log.gz 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34765.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35068.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34856.log.gz 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34765.log.gz 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34897.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.35068.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:38:37.692 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.34897.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35068.log.gz 2024-03-25T18:38:37.693 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34979.log 2024-03-25T18:38:37.693 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:38:37.693 INFO:teuthology.orchestra.run.smithi146.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34897.log.gz 2024-03-25T18:38:37.693 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.34979.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34979.log.gz 2024-03-25T18:38:37.693 INFO:teuthology.orchestra.run.smithi146.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:38:37.693 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34937.log 2024-03-25T18:38:37.693 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34937.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34937.log.gz 2024-03-25T18:38:37.699 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41351.log 2024-03-25T18:38:37.699 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40177.log 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35709.log 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41351.log.gz 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39592.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40177.log.gz 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.35709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40803.log 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39904.log 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35709.log.gz 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41975.log 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41194.log 2024-03-25T18:38:37.700 INFO:teuthology.orchestra.run.smithi102.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40803.log.gz 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40374.log 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39904.log.gz 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41819.log 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41975.log.gz 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40374.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41194.log.gz 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40725.log 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40413.log 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40374.log.gz 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40843.log 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41819.log.gz 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40725.log.gz 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39122.log 2024-03-25T18:38:37.701 INFO:teuthology.orchestra.run.smithi102.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40413.log.gz 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40216.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40843.log.gz 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39122.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39122.log.gz 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:38:37.702 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37666.log 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39358.log 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr: 72.0%/var/log/ceph/ceph-client.admin.37666.log: -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41312.log 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40569.log 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37666.log.gz 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36612.log 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39358.log.gz 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40335.log 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41312.log.gz 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.36612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35560.log 2024-03-25T18:38:37.703 INFO:teuthology.orchestra.run.smithi102.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40569.log.gz 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log/var/log/ceph/ceph-client.admin.40335.log: 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36612.log.gz 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.35560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40335.log.gz 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42131.log 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35560.log.gz 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41897.log 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41702.log: /var/log/ceph/ceph-client.admin.42131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gz 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41038.log 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42131.log.gz 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39943.log 2024-03-25T18:38:37.704 INFO:teuthology.orchestra.run.smithi102.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41897.log.gz 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35231.log 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41273.log 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41038.log.gz 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.35231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41155.log 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39943.log.gz 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42209.log 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41273.log: /var/log/ceph/ceph-client.admin.41155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41429.log 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35231.log.gz 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42209.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41273.log.gz 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41780.log 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41155.log.gz 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-03-25T18:38:37.705 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41429.log: /var/log/ceph/ceph-client.admin.41780.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42209.log.gz 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41429.log.gz 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42092.log 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41780.log.gz 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40452.log 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.audit.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41663.log 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40452.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42092.log.gz 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41936.log 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40138.log 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40452.log.gz 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40530.log 2024-03-25T18:38:37.706 INFO:teuthology.orchestra.run.smithi102.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41663.log.gz 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41233.log 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41936.log.gz 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39865.log 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40138.log.gz 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40021.log 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40530.log.gz 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40099.log 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41233.log.gz 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40491.log 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39865.log.gz 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40882.log 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40021.log.gz 2024-03-25T18:38:37.707 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40099.log.gz 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40491.log.gz 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42014.log 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40882.log.gz 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40686.log 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2024-03-25T18:38:37.708 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42014.log.gz 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39280.log 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40921.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40686.log.gz 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41468.log 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35537.log 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39280.log: /var/log/ceph/ceph-client.admin.41468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41624.log 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.35537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40060.log 86.9% 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39280.log.gz 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41468.log.gz 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41624.log: 85.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.35537.log.gz -5 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39826.log 2024-03-25T18:38:37.709 INFO:teuthology.orchestra.run.smithi102.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41624.log.gz 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39202.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40060.log.gz 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39787.log 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39826.log: /var/log/ceph/ceph-osd.3.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39826.log.gz 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39397.log 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42248.log 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39787.log.gz 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40647.log 2024-03-25T18:38:37.710 INFO:teuthology.orchestra.run.smithi102.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39631.log 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42248.log.gz 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39436.log 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39631.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40647.log.gz 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42287.log 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39436.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39631.log.gz 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39319.log 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz/var/log/ceph/ceph-client.admin.42287.log: 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40608.log 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39319.log: gzip -5 --verbose -- 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42287.log.gz 2024-03-25T18:38:37.711 INFO:teuthology.orchestra.run.smithi102.stderr: /var/log/ceph/ceph-client.admin.40960.log 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40608.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39241.log 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40960.log: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40608.log.gz 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41585.log 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40960.log.gz 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40255.log 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39241.log.gz 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29377.log 2024-03-25T18:38:37.712 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41585.log: /var/log/ceph/ceph-client.admin.40255.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41585.log.gz 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40255.log.gz 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39083.log 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.tmp-client.admin.29377.log: gzip -5 0.0% --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.29377.log.gz 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39083.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose/var/log/ceph/ceph.log: -- 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39083.log.gz 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr: /var/log/ceph/ceph-client.admin.41546.log 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41546.log: /var/log/ceph/ceph-osd.2.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41546.log.gz 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39982.log 2024-03-25T18:38:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39514.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41116.log 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39163.log 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39982.log.gz 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40999.log 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39163.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41116.log.gz 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39163.log.gz 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42365.log 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40296.log 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40999.log: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-03-25T18:38:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40296.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40999.log.gz 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42365.log.gz 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42326.log 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40296.log.gz 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41858.log 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.38873.log: /var/log/ceph/ceph-client.admin.42326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42170.log 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41858.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42326.log.gz 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42053.log 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41858.log.gz 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42170.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42170.log.gz 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42053.log.gz 2024-03-25T18:38:37.715 INFO:teuthology.orchestra.run.smithi102.stderr: /var/log/ceph/ceph-client.admin.41741.log 2024-03-25T18:38:37.716 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.39748.log: /var/log/ceph/ceph-client.admin.41741.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2024-03-25T18:38:37.716 INFO:teuthology.orchestra.run.smithi102.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41741.log.gz 2024-03-25T18:38:37.749 INFO:teuthology.orchestra.run.smithi146.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:38:38.528 INFO:teuthology.orchestra.run.smithi146.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:38:38.566 INFO:teuthology.orchestra.run.smithi146.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:38:38.576 INFO:teuthology.orchestra.run.smithi146.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:38:38.885 INFO:teuthology.orchestra.run.smithi102.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:38:38.894 INFO:teuthology.orchestra.run.smithi146.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:38:38.902 INFO:teuthology.orchestra.run.smithi102.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:38:38.913 INFO:teuthology.orchestra.run.smithi102.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:38:38.922 INFO:teuthology.orchestra.run.smithi102.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:38:42.498 INFO:teuthology.orchestra.run.smithi102.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:38:42.809 INFO:teuthology.orchestra.run.smithi102.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:38:42.811 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-25T18:38:42.811 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m5.187s 2024-03-25T18:38:42.811 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m14.529s 2024-03-25T18:38:42.811 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m0.575s 2024-03-25T18:38:42.883 INFO:teuthology.orchestra.run.smithi146.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:38:42.884 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-25T18:38:42.884 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m5.229s 2024-03-25T18:38:42.885 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m8.850s 2024-03-25T18:38:42.885 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.348s 2024-03-25T18:38:42.886 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:38:42.886 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/remote/smithi102/log 2024-03-25T18:38:42.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:38:45.249 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/remote/smithi146/log 2024-03-25T18:38:45.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:38:46.845 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:38:47.077 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=147760a0a14348e0a59c47293c82ed0b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:38:47.082 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:38:47.092 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:38:47.099 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:38:47.099 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:38:47.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:38:47.200 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:38:47.208 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:38:47.209 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:38:47.211 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:38:47.224 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:38:47.228 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:38:47.230 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:38:47.230 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-25T18:38:47.230 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 59 +522us[ +522us] +/- 76ms 2024-03-25T18:38:47.231 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -1077us[-1097us] +/- 61ms 2024-03-25T18:38:47.231 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +773us[ +773us] +/- 66ms 2024-03-25T18:38:47.231 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:38:47.233 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:38:47.233 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-25T18:38:47.233 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +529us[ +510us] +/- 76ms 2024-03-25T18:38:47.233 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -1078us[-1103us] +/- 62ms 2024-03-25T18:38:47.233 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +771us[ +752us] +/- 66ms 2024-03-25T18:38:47.233 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:38:47.234 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:38:47.243 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:38:47.243 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:38:47.251 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:38:47.339 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:38:47.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:38:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:38:47.442 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:38:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:38:47.473 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:38:47.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:38:47.491 INFO:teuthology.task.internal:Duration was 7831.057529 seconds 2024-03-25T18:38:47.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:38:47.500 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:38:47.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:38:47.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:38:47.544 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:38:47.558 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:38:47.764 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:38:47.765 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-03-25T18:38:47.765 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:38:47.797 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-03-25T18:38:47.797 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:38:47.823 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:38:47.823 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:38:47.840 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:38:47.892 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:38:47.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:38:47.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:38:48.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:38:48.047 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:38:48.047 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:38:48.074 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:38:48.100 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:38:48.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:38:48.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:38:48.139 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-03-25T18:38:48.166 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-03-25T18:38:48.211 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:38:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:38:48.225 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:38:48.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:38:48.266 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:38:48.275 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:38:48.276 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/remote/smithi102 2024-03-25T18:38:48.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:38:48.311 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280/remote/smithi146 2024-03-25T18:38:48.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:38:48.347 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:38:48.347 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:38:48.350 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:38:48.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:38:48.408 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:38:48.408 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:38:48.416 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:38:48.416 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:38:48.419 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:38:48.431 INFO:teuthology.orchestra.run.smithi102.stdout: 266033 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:38 /home/ubuntu/cephtest 2024-03-25T18:38:48.450 INFO:teuthology.orchestra.run.smithi146.stdout: 265983 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:38 /home/ubuntu/cephtest 2024-03-25T18:38:48.451 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:38:48.460 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:38:48.478 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:38:48.503 DEBUG:teuthology.nuke:shortname: smithi102 2024-03-25T18:38:48.504 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:38:48.528 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:19:17.529385', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBAYvnjUY3nI+HIYO8Dn/SgNRNe9ZigdnjOdZ4vy2p9LJP7xZ4RUQ3G8wxeCTNr4a7yQkLUGc2eFGGr0tXpEf8='} 2024-03-25T18:38:48.531 DEBUG:teuthology.nuke:shortname: smithi146 2024-03-25T18:38:48.532 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:38:48.557 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621280', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:19:17.526797', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDmwdMj0KKr+QmizJqJ9GC8PcQVl5kT7qN2x5x7cetOt//LXCqLl6za4k2jkJoSSdf8yvXr3HCWWda3EgwHDTI8='} 2024-03-25T18:38:48.582 INFO:teuthology.orchestra.console.smithi102:Power off 2024-03-25T18:38:48.582 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:38:48.606 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-03-25T18:38:48.606 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:38:48.622 INFO:teuthology.orchestra.console.smithi146:Power off 2024-03-25T18:38:48.623 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:38:48.632 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-25T18:38:48.645 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-03-25T18:38:48.645 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:38:48.666 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-25T18:38:52.633 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:38:52.668 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:38:52.744 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-25T18:38:52.778 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-25T18:38:56.750 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:38:56.779 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:38:56.861 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-25T18:38:56.888 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-25T18:39:00.865 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:39:00.889 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:39:00.975 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-03-25T18:39:00.999 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-03-25T18:39:01.076 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-03-25T18:39:01.100 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-03-25T18:39:01.206 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7831.057529449463 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=147760a0a14348e0a59c47293c82ed0b status: fail success: false 2024-03-25T18:39:01.207 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:39:01.284 INFO:teuthology.run:FAIL